This is a merge of pre-trained language models created using
mergekit.
This model essentially a rebuild of v1, using task arithmetic instead of SLERP.
Of note, Mistral Nemo Base was used as the base model for task arithmetic, rather than Instruct or another fine-tune.
Furthermore, max_position_embeddings was reduced to 131072, down from 1000000, as the model was only trained on up to 131072.
Tested with temperature 1.0 and minP 0.01; temperature can be reduced (briefly tested at 0.45) if the model is too aggressively creative/hallucinatory.
This model was merged using the
task arithmetic merge method using
grimjim/mistralai-Mistral-Nemo-Base-2407 as a base.
1base_model: grimjim/mistralai-Mistral-Nemo-Base-2407
2dtype: bfloat16
3merge_method: task_arithmetic
4parameters:
5 normalize: false
6slices:
7- sources:
8 - layer_range: [0, 40]
9 model: grimjim/mistralai-Mistral-Nemo-Base-2407
10 - layer_range: [0, 40]
11 model: grimjim/mistralai-Mistral-Nemo-Instruct-2407
12 parameters:
13 weight: 0.9
14 - layer_range: [0, 40]
15 model: grimjim/magnum-consolidatum-v1-12b
16 parameters:
17 weight: 0.1
18
