Simple merges of my diffrent models, then finetuned for 500 steps using lora and oasst1.
This is a merge of pre-trained language models created using
mergekit.
This model was merged using the Passthrough merge method.
1slices:
2 # oasst2-llama repeated 3×
3 - sources:
4 - model: simonko912/oasst2-llama
5 layer_range: [0, 12]
6 - sources:
7 - model: simonko912/oasst2-llama
8 layer_range: [0, 12]
9 - sources:
10 - model: simonko912/oasst2-llama
11 layer_range: [0, 12]
12
13 # oasst-big-llama
14 - sources:
15 - model: simonko912/oasst-big-llama
16 layer_range: [0, 12]
17
18 # oasst1-llama
19 - sources:
20 - model: simonko912/oasst1-llama
21 layer_range: [0, 12]
22
23merge_method: passthrough
24dtype: bfloat16
25
