This was a test of mine how small merges perform, because there are a lot of 7b merges and higher but not a lot of 2b merges.
This model was merged using the SLERP merge method.
1slices:
2- sources:
3 - model: Weyaxi/Einstein-v4-phi2
4 layer_range:
5 - 0
6 - 32
7 - model: rhysjones/phi-2-orange-v2
8 layer_range:
9 - 0
10 - 32
11merge_method: slerp
12base_model: rhysjones/phi-2-orange-v2
13parameters:
14 t:
15 - filter: self_attn
16 value:
17 - 0
18 - 0.5
19 - 0.3
20 - 0.7
21 - 1
22 - filter: mlp
23 value:
24 - 1
25 - 0.5
26 - 0.7
27 - 0.3
28 - 0
29 - value: 0.5
30dtype: bfloat16
Use it with the ChatML format, you can also use the Inference-API for this Model.
Detailed results can be found
here
