A merge of several Yi 34B 200K models using the new DARE Ties method via mergekit. The goal is to create a merge model that excels at 32K+ context performance.

SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:

It might recognize ChatML, and possibly Alpaca-like formats. Raw prompting as described here is also effective: https://old.reddit.com/r/LocalLLaMA/comments/18zqy4s/the_secret_to_writing_quality_stories_with_llms/

Being a Yi model, try running a lower temperature with 0.02-0.06 MinP, a little repetition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default, and it really needs a low temperature + MinP to cull the huge vocabulary.

24GB GPUs can efficiently run Yi-34B-200K models at 45K-90K context with exllamav2, and performant UIs like exui. I go into more detail in this post. 16GB GPUs can still run the high context with aggressive quantization.

To load/train this in full-context backends like transformers, you must change max_position_embeddings in config.json to a lower value than 200,000, otherwise you will OOM! I do not recommend running high context without context-efficient backends like exllamav2 or unsloth.

See: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-merge-v5#testing-notes

A "4k" merge model was created to try and extend the context of SUS Chat and DPO-bagel before adding them to the merge: https://huggingface.co/brucethemoose/SUS-Bagel-200K-DARE-Test

In addition, the weight gradients are biased towards Vicuna-format models in the first few layers to try and "emphasize" the Orca-Vicuna prompt template. How sucessful this is remains to be seen.

This model was merged using the DARE TIES merge method using /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama as a base.

The following models were included in the merge:

The following YAML configuration was used to produce this model:

The following config was used for the "4kmerge" model:

Detailed results can be found here