An attempt to make a functional goliath style merge to create a [Etheria] 55b-200k with two yi-34b-200k models.
due to the merge it 'theoretically' should have a context of 200k but I recommend starting at 32k and moveing up,
as it is unknown (at this time) what the merge has done to the context length.
This is a merge of both VerA and VerB of Etheria-55b (There numbers were surprisingly good), I then created a sacrificial 55B out of the most performant yi-34b-200k Model
and performed a Dare_ties merge and equalize the model into its current state.
This model was merged using the
DARE TIES merge method using Merged-Etheria-55b as a base.
1
2base_model: Merged-Etheria-55b
3models:
4 - model: Sacr-Etheria-55b
5 parameters:
6 weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
7 density: 0.61
8 - model: Merged-Etheria-55b
9 parameters:
10 weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
11 density: 0.61
12merge_method: dare_ties
13tokenizer_source: union
14parameters:
15 int8_mask: true
16dtype: bfloat16
17
