This merge amounts to the grafting of a model derived from Mistral v0.1 7B (4K sliding window context, to a maximum of 8K practical context length) onto a model derived from Mistral v0.2 7B (32K context length, to a maximum of 16K practical context length).

Code from the Mistral Python script extend_model_vocab.py was used to extend the vocabulary from size 32000 to 32768, adapted to work on safetensor shards. The resulting model appears to be broken, unfortunately. However, this opens up the model to unconventional merger with Mistral v0.3 7B models.

This was a merge of pre-trained language models created using mergekit.

This model was merged using the task arithmetic merge method using grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B as a base.

The following models were included in the merge:

The following YAML configuration was used to produce this model: