This is a merge of pre-trained language models created using
mergekit.
I Greatly expanded the amount of models used in this merge, experimented a lot with different idea's.
This version feels a lot more convincing than V1.5 Hopefully the long context window will also remain strong after Quants.
Because of the many merges switched back from BFloat to Float.
Tried breadcrums without the Ties, that went very poorly.
This model was merged using the breadcrumbs_ties merge method using I:\Llama-3-70B-Instruct-Gradient-262k as a base.
1models:
2 - model: \Llama-3-70B-Instruct-Gradient-262k
3 parameters:
4 weight: 0.25
5 density: 0.90
6 gamma: 0.01
7 - model: \Meta-LLama-3-Cat-Smaug-LLama-70b
8 parameters:
9 weight: 0.28
10 density: 0.90
11 gamma: 0.01
12 - model: \Llama-3-Lumimaid-70B-v0.1-alt
13 parameters:
14 weight: 0.15
15 density: 0.90
16 gamma: 0.01
17 - model: \Tess-2.0-Llama-3-70B-v0.2
18 parameters:
19 weight: 0.06
20 density: 0.90
21 gamma: 0.01
22 - model: \Smaug-Llama-3-70B-Instruct
23 parameters:
24 weight: 0.04
25 density: 0.90
26 gamma: 0.01
27 - model: \Llama-3-70B-Synthia-v3.5
28 parameters:
29 weight: 0.05
30 density: 0.90
31 gamma: 0.01
32 - model: \Llama-3-70B-Instruct-Gradient-524k
33 parameters:
34 weight: 0.03
35 density: 0.90
36 gamma: 0.01
37 - model: \Meta-LLama-3-Cat-A-LLama-70b
38 parameters:
39 weight: 0.14
40 density: 0.90
41 gamma: 0.01
42merge_method: breadcrumbs_ties
43base_model: I:\Llama-3-70B-Instruct-Gradient-262k
44dtype: float16
