This is an experimental merge, replicating additional layers to the model without post-merge healing.
There is damage to the model, but it appears to be tolerable as is; the performance difference in benchmarks from the original 8B Instruct model does not appear to be significant.
The resulting impact on narrative text completion may also be of interest.
Light testing performed with instruct prompting and the following sampler settings:
This is a merge of pre-trained language model meta-llama/Meta-Llama-3-8B-Instruct created using
mergekit.
Built with Meta Llama 3.
This model was merged using the passthrough merge method.
1slices:
2 - sources:
3 - model: meta-llama/Meta-Llama-3-8B-Instruct
4 layer_range: [0, 12]
5 - sources:
6 - model: meta-llama/Meta-Llama-3-8B-Instruct
7 layer_range: [8, 32]
8merge_method: passthrough
9dtype: bfloat16
10
