This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
Linear DELLA merge method using
Vortex5/MegaMoon-Karcher-12B as a base.
1
2models:
3 - model: Vortex5/Vermilion-Sage-12B
4 parameters:
5 weight: [0.2, 0.5, 0.9, 1.0, 0.95, 0.8, 0.4, 0.2]
6 density: 0.55
7 epsilon: 0.4
8 - model: Vortex5/Dark-Quill-12B
9 parameters:
10 weight: [0.8, 1.0, 0.9, 0.7, 0.5, 0.4, 0.2, 0.0]
11 density: 0.5
12 epsilon: 0.4
13merge_method: della_linear
14base_model: Vortex5/MegaMoon-Karcher-12B
15parameters:
16 lambda: 0.94
17 normalize: true
18dtype: bfloat16
19tokenizer:
20 source: union
21
