Meta's Llama 3 8B pruned to 7B parameters(w/ 28 layers). Layers to prune selected using PruneMe repo on Github.
-
layers_to_skip = 4
-
Layer 23 to 27 has the minimum average distance of 0.18376044921875
-
To Do : Post pruning training.
This is a merge of pre-trained language models created using
mergekit.
This model was merged using the passthrough merge method.
1slices:
2 - sources:
3 - model: meta-llama/Meta-Llama-3-8B
4 layer_range: [0, 23]
5 - sources:
6 - model: meta-llama/Meta-Llama-3-8B
7 layer_range: [27,32]
8
9merge_method: passthrough
10dtype: bfloat16
11
