This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
SLERP merge method.
1slices:
2
3 - sources:
4
5 - model: Tarek07/Progenitor-V1.1-LLaMa-70B
6
7 layer_range: [0, 32]
8
9 - model: TareksLab/Thalassic-Alpha-LLaMa-70B
10
11 layer_range: [0, 32]
12
13merge_method: slerp
14
15base_model: Tarek07/Progenitor-V1.1-LLaMa-70B
16
17parameters:
18
19 t:
20
21 - filter: self_attn
22
23 value: [0, 0.5, 0.3, 0.7, 1]
24
25 - filter: mlp
26
27 value: [1, 0.5, 0.7, 0.3, 0]
28
29 - value: 0.5
30
31dtype: bfloat16
