Currently the other versions resulted in really bad results that I didn't upload them, the version number is just the internal version.
The idea is to keep the strengths of
anthracite-org/magnum-12b-v2 while adding some more creativity
that seems to be lacking in the model. Mistral-Nemo by itself seems to behave less sporadic due to the low temperature needed but this gets a bit repetitive,
although it's still the best model I've used so far.
I am not entirely pleased with the result of the merge but it seems okay, though base
anthracite-org/magnum-12b-v2
might just be better by itself. However, I'll still experiement on different merge methods.
Leaking of the training data used on both models seems a bit more apparent when using higher temperature values,
especially the use of author notes on the system prompt. Generally I'd advise to create a stopping string for "```" to avoid the generation of the training data.
Original Models:This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
TIES merge method using models/magnum-12b-v2 as a base.
1models:
2 - model: anthracite-org/magnum-12b-v2
3 parameters:
4 density: 0.3
5 weight: 0.5
6 - model: UsernameJustAnother/Nemo-12B-Marlin-v5
7 parameters:
8 density: 0.7
9 weight: 0.5
10merge_method: ties
11base_model: anthracite-org/magnum-12b-v2
12parameters:
13 normalize: true
14 int8_mask: true
15dtype: bfloat16
