This merge is successful. Not adding or editorializing the model card right now. I need sleep. But, resultant model works great! This experiment revealed two things. One, distilled instruct models work best for TIES merging with the base and other models; the experiment showed that this is due to the way that distilled models are trained vs non-distilled models: when merged with other models, the distilled models seem to retain more of their attributes (the way that they talk, think, reason, etc) - this makes them very appealing for model merges because you keep more of the model's inherent capabilities and behaviors. And, two: I can successfully TIES merge different instruct models with their base pre-initialized to the embeddings special tokens (for prompt/chat template).
The model is coherent and capable. Please download and try it if your interested. GGUF Custom OQ8_0-F32_EF32 IQuants will be up by the middle of the week - most probably sooner but still...
This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
TIES merge method using /Users/jsarnecki/opt/mergekit/merges/Llama-3.1-8B-InitializedEmbeddings_with_Hermes-3 as a base.
1models:
2
3 - model: "/Users/jsarnecki/opt/Workspace/arcee-ai/Llama-3.1-SuperNova-Lite"
4 parameters:
5 weight: 1
6 density: 1
7
8 - model: "/Users/jsarnecki/opt/Workspace/NousResearch/Hermes-3-Llama-3.1-8B"
9 parameters:
10 weight: 1
11 density: 1
12
13 - model: "/Users/jsarnecki/opt/Workspace/arcee-ai/Llama-3.1-SuperNova-Lite"
14 parameters:
15 weight: 1
16 density: 1
17
18 - model: "/Users/jsarnecki/opt/Workspace/NousResearch/Hermes-3-Llama-3.1-8B"
19 parameters:
20 weight: 1
21 density: 1
22
23merge_method: ties
24base_model: "/Users/jsarnecki/opt/mergekit/merges/Llama-3.1-8B-InitializedEmbeddings_with_Hermes-3"
25parameters:
26 density: 1
27 normalize: true
28 int8_mask: true
29tokenizer_source: "/Users/jsarnecki/opt/Workspace/NousResearch/Hermes-3-Llama-3.1-8B"
30dtype: float32
31out_dtype: bfloat16
32
