This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
linear merge method using
Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small as a base.
1slices:
2 - sources:
3 - model: TheDrummer/UnslopSmall-22B-v1
4 layer_range: [0, 32]
5 - model: Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small
6 layer_range: [0, 32]
7merge_method: linear
8base_model: Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small
9parameters:
10 weight:
11 - filter: self_attn
12 value: [0.2, 0.8]
13 - filter: mlp
14 value: [0.8, 0.2]
15 - value: 0.5
16dtype: bfloat16
