This is a merge of pre-trained language models created using
mergekit.
This model was merged using the
SLERP merge method.
1base_model: tinycompany/Shawty-1.4B-SFT-Stage-1
2merge_method: slerp
3tokenizer_source: base
4dtype: bfloat16
5parameters:
6 t:
7 - filter: self_attn
8 value: [0.2, 0.6, 0.4, 0.8, 1.0]
9 - filter: mlp
10 value: [1.0, 0.6, 0.8, 0.4, 0.2]
11 - value: 0.3
12slices:
13 - sources:
14 - model: tinycompany/Shawty-1.4B-SFT-Stage-1
15 layer_range: [0, 28]
16 - model: tinycompany/shawty-CoT-Hindi-English
17 layer_range: [0, 28]
18
