This model benchmarks quite well compared to other 7b models, and has exceptional
MT-Bench and
EQ-Bench v2.1 scores, ranking higher than ChatGPT-3.5-turbo and Claude-1 in both tests, and Goliath-120b, and other 70B models in the latter .
This is a merge of pre-trained language models created using
mergekitThis model was merged using the
DARE TIES merge method using
mistralai/Mistral-7B-v0.1 as a base.
Density was chosen deterministically between the models chosen for this merge. After testing many densities, I settled on 0.58 for each of the chosen models as it returned the highest EQ-Bench score. Not much testing was done with the weights, but I thought that I'd try gradients. Conceptually, Westlake and a Distilled version of Open Heremes are heavier in the initial layers (guiding understanding, and thoughts), before Noromaid and AlphaMonarch come in to guide its wants, reasoning, and conversation.
1models:
2 - model: mistralai/Mistral-7B-v0.1
3 # No parameters necessary for base model
4 - model: senseable/WestLake-7B-v2
5 parameters:
6 density: 0.58
7 weight: [0.50, 0.40, 0.25, 0.05]
8 - model: NeverSleep/Noromaid-7B-0.4-DPO
9 parameters:
10 density: 0.58
11 weight: [0.05, 0.05, 0.25, 0.40]
12 - model: argilla/distilabeled-OpenHermes-2.5-Mistral-7B
13 parameters:
14 density: 0.58
15 weight: [0.40, 0.50, 0.25, 0.05]
16 - model: mlabonne/AlphaMonarch-7B
17 parameters:
18 density: 0.58
19 weight: [0.05, 0.05, 0.25, 0.50]
20merge_method: dare_ties
21base_model: mistralai/Mistral-7B-v0.1
22parameters:
23 int8_mask: true
24dtype: bfloat16
25
