NemoMix-12B-DellaV1a is an
experimental merge of the following models using the
DELLA method with
mergekit:
EDIT: There seem to be tokenizer issues. I would probably have to merge with the base model instead of instruct I'm guessing. Don't bother.
1models:
2 - model: BeaverAI/mistral-doryV2-12b
3 parameters:
4 weight: 0.20
5 density: 0.42
6 - model: NeverSleep/Lumimaid-v0.2-12B
7 parameters:
8 weight: 0.22
9 density: 0.54
10 - model: intervitens/mini-magnum-12b-v1.1
11 parameters:
12 weight: 0.24
13 density: 0.66
14 - model: grimjim/mistralai-Mistral-Nemo-Instruct-2407
15 parameters:
16 weight: 0.34
17 density: 0.78
18merge_method: della
19base_model: grimjim/mistralai-Mistral-Nemo-Instruct-2407
20parameters:
21 int8_mask: true
22 epsilon: 0.1
23 lambda: 1.0
24 density: 0.7
25dtype: bfloat16
26
