This is a merge of pre-trained language models created using
mergekit.
This model was merged using the task_swapping_ties merge method using
ammarali32/multi_verse_model as a base.
1base_model:
2 model:
3 path: ammarali32/multi_verse_model
4dtype: bfloat16
5merge_method: task_swapping_ties
6slices:
7- sources:
8 - layer_range: [0, 32]
9 model:
10 model:
11 path: yam-peleg/Experiment26-7B
12 parameters:
13 diagonal_offset: 2.0
14 weight: 0.4
15 - layer_range: [0, 32]
16 model:
17 model:
18 path: ammarali32/multi_verse_model
19 parameters:
20 weight: 0.6
