You can deploy this model and start earning money today!

brucethemoose/
CapyTessBorosYi-34B-200K-DARE-Ties-exl2-3.1bpw-fiction

Name: CapyTessBorosYi-34B-200K-DARE-Ties-exl2-3.1bpw-fiction
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

NousResearch/Nous-Capybara-34B, migtissera/Tess-M-v1.3 and bhenrym14/airoboros-3_1-yi-34b-200k merged with a new, experimental implementation of "dare ties" via mergekit.

Qantized with exllamav2 on 200 rows (400K tokens) on a long Orca-Vicuna format chat, a sci fi story and a fantasy story. This should hopefully yield better chat performance than the default wikitext quantization.

3.1bpw is enough for ~75K context on a 24GB GPU.. I would highly recommend running in exui for speed at long context, I go into more detail in this Reddit post

Merged with the following config, and the tokenizer from chargoddard's Yi-Llama:

models:
  - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
    # no parameters necessary for base model
  - model: /home/alpha/Storage/Models/Raw/migtissera_Tess-M-v1.3
    parameters:
      weight: 0.41
      density: 0.50
  - model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
    parameters:
      weight: 0.18
      density: 0.46
  - model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
    parameters:
      weight: 0.41
      density: 0.50
merge_method: dare_ties
base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
parameters:
int8_mask: true
dtype: bfloat16

First exllama quantization pass:

python convert.py --in_dir //home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/capytessborosmes.json --cal_dataset /home/alpha/Documents/smol.parquet -l 2048 -r 80 -ml 2048 -mr 40 -gr 40 -ss 4096 -nr -b 4.0 -hb 6

Second exllama quantization pass:

python convert.py --in_dir /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/capytessmes.json --cal_dataset /home/alpha/Documents/medium.parquet -l 2048 -r 200 -ml 2048 -mr 40 -gr 200 -ss 4096 -b 3.1 -hb 6 -cf /home/alpha/FastModels/CapyTessBorosYi-34B-200K-DARE-Ties-exl2-4bpw-fiction -nr

dare_ties is testing with better perplexity than a regular ties merge with the same merge configuration. Model weights that add up to one also seem optimal from testing. And results at long context seem... better than the previous dare merge with Tess 1.2?

I chose not to include other finetunes, such as Dolphin, because they aren't trained on the 200K base. If any other 200K finetunes pop up, let me know.

Prompt template: Orca-Vicuna

SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:

Being a Yi model, try disabling the BOS token and/or running a lower temperature with MinP (and no other samplers) if output doesn't seem right. Yi tends to run "hot" by default.

Sometimes the model "spells out" the stop token as </s> like Capybara, so you may need to add </s> as an additional stopping condition. It also might respond to the llama-2 chat format.

Credits: https://github.com/turboderp/exllamav2

https://github.com/cg123/mergekit/tree/dare

https://huggingface.co/NousResearch/Nous-Capybara-34B/

https://huggingface.co/bhenrym14/airoboros-3_1-yi-34b-200k

https://huggingface.co/migtissera/Tess-M-v1.3

https://huggingface.co/chargoddard/Yi-34B-200K-Llama

https://huggingface.co/01-ai/Yi-34B-200K

brucethemoose/CapyTessBorosYi-34B-200K-DARE-Ties-exl2-3.1bpw-fiction

Prompt template: Orca-Vicuna

brucethemoose/
CapyTessBorosYi-34B-200K-DARE-Ties-exl2-3.1bpw-fiction