This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
SLERP merge method.
1base_model: mistralai/Mistral-7B-v0.1
2dtype: float16
3merge_method: slerp
4parameters:
5 t:
6 - filter: self_attn
7 value:
8 - 0
9 - 0.5
10 - 0.3
11 - 0.7
12 - 1
13 - filter: mlp
14 value:
15 - 1
16 - 0.5
17 - 0.7
18 - 0.3
19 - 0
20 - value: 0.5
21slices:
22- sources:
23 - layer_range:
24 - 0
25 - 32
26 model: mistralai/Mistral-7B-v0.1
27 - layer_range:
28 - 0
29 - 32
30 model: Open-Orca/Mistral-7B-OpenOrca
31
