Credit for the model card's description goes to ddh0 and mergekit
Credit for access and conversion of Mistral-7B-v0.2 goes to alpindale (from MistalAI's weights to HF Transformers)
Mistral-12.25B-v0.2
This is Mistral-12.25B-v0.2, a depth-upscaled version of
alpindale/Mistral-7B-v0.2-hf.
This model is intended to be used as a basis for further fine-tuning, or as a drop-in upgrade from the original 7 billion parameter model.
This is a merge of pre-trained language models created using
mergekit.
UpStage's conclusionary limitations of their research:
"Our study on the Depth Up-Scaling (DUS) has important limitations and considerations. One key limitation is the need for more thorough explorations of hyperparameters used in the DUS approach. Namely, we removed m = 8 layers from both ends of our base model, primarily due to hardware limitations. However, we have not yet determined if this value is optimal for enhancing performance. The extended time and cost of continued pretraining made it challenging to conduct more comprehensive experiments, which we aim to address in future work through various comparative analyses."
This model was made to help test whether 10.7B parameters (m = 8) is better or worse than m < 8 (10.7B+ parameters)
Merge Details
Merge Method
This model was merged using the passthrough merge method.
Models Merged
The following models were included in the merge:
- /Users/jsarnecki/opt/Workspace/alpindale/Mistral-7B-v0.2-hf
Configuration
The following YAML configuration was used to produce this model:
1dtype: bfloat16
2merge_method: passthrough
3# Depth UpScaled (DUS) version of Mistral-7B-v0.2
4# where m = 4 (The number of layers to remove from the model)
5# s = 56 (The number of layers the model will have after the DUS)
6slices:
7- sources:
8 - layer_range: [0, 28]
9 model: /Users/jsarnecki/opt/Workspace/alpindale/Mistral-7B-v0.2-hf
10- sources:
11 - layer_range: [4, 32]
12 model: /Users/jsarnecki/opt/Workspace/alpindale/Mistral-7B-v0.2-hf
13
exllama (Thanks to blockblockblock)
https:// huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw2.5
https://huggingtace.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw3
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw3.5
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw3.7
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw4
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw4.2
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw4.4
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw4.6
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw4.8
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw5
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-vO.2-bpw5.5
https://huggingface.co/blockblockblock/
Mistral-12.25B-Instruct-v0.2-bpw6
