INEX4-7b is a merge of the following models using
mergekit:
1slices:
2 - sources:
3 - model: liminerity/Ingot-7b-slerp-7-forged
4 layer_range: [0, 32]
5 - model: yam-peleg/Experiment26-7B
6 layer_range: [0, 32]
7merge_method: slerp
8base_model: liminerity/Ingot-7b-slerp-7-forged
9parameters:
10 t:
11 - filter: self_attn
12 value: [0, 0.5, 0.3, 0.7, 1]
13 - filter: mlp
14 value: [1, 0.5, 0.7, 0.3, 0]
15 - value: 0.5
16dtype: bfloat16
17model: liminerity/merge
18
19slices:
20 - sources:
21 - model: liminerity/Ingot-7b-slerp-7-forged
22 layer_range: [0, 32]
23 - model: liminerity/merge
24 layer_range: [0, 32]
25merge_method: slerp
26base_model: liminerity/Ingot-7b-slerp-7-forged
27parameters:
28 t:
29 - filter: self_attn
30 value: [0, 0.5, 0.3, 0.7, 1]
31 - filter: mlp
32 value: [1, 0.5, 0.7, 0.3, 0]
33 - value: 0.5
34dtype: bfloat16
35model: liminerity/merge1
36
37slices:
38 - sources:
39 - model: yam-peleg/Experiment26-7B
40 layer_range: [0, 32]
41 - model: liminerity/merge1
42 layer_range: [0, 32]
43merge_method: slerp
44base_model: yam-peleg/Experiment26-7B
45parameters:
46 t:
47 - filter: self_attn
48 value: [0, 0.5, 0.3, 0.7, 1]
49 - filter: mlp
50 value: [1, 0.5, 0.7, 0.3, 0]
51 - value: 0.5
52dtype: float16
53model: liminerity/merge2
54
55slices:
56 - sources:
57 - model: liminerity/merge2
58 layer_range: [0, 32]
59 - model: liminerity/merge1
60 layer_range: [0, 32]
61merge_method: slerp
62base_model: liminerity/merge2
63parameters:
64 t:
65 - filter: self_attn
66 value: [0, 0.5, 0.3, 0.7, 1]
67 - filter: mlp
68 value: [1, 0.5, 0.7, 0.3, 0]
69 - value: 0.5
70dtype: bfloat16
71model: INEX-7b
Detailed results can be found
here
