You can deploy this model and start earning money today!

RichardErkhov/
brucethemoose_-_Yi-34B-200K-DARE-merge-v7-gguf

Name: brucethemoose_-_Yi-34B-200K-DARE-merge-v7-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Yi-34B-200K-DARE-merge-v7 - GGUF

Model creator: https://huggingface.co/brucethemoose/
Original model: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-merge-v7/

Name	Quant method	Size
Yi-34B-200K-DARE-merge-v7.Q2_K.gguf	Q2_K	11.94GB
Yi-34B-200K-DARE-merge-v7.IQ3_XS.gguf	IQ3_XS	0.65GB
Yi-34B-200K-DARE-merge-v7.IQ3_S.gguf	IQ3_S	0.24GB
Yi-34B-200K-DARE-merge-v7.Q3_K_S.gguf	Q3_K_S	13.93GB
Yi-34B-200K-DARE-merge-v7.IQ3_M.gguf	IQ3_M	14.5GB
Yi-34B-200K-DARE-merge-v7.Q3_K.gguf	Q3_K	15.51GB
Yi-34B-200K-DARE-merge-v7.Q3_K_M.gguf	Q3_K_M	15.51GB
Yi-34B-200K-DARE-merge-v7.Q3_K_L.gguf	Q3_K_L	16.89GB
Yi-34B-200K-DARE-merge-v7.IQ4_XS.gguf	IQ4_XS	2.28GB
Yi-34B-200K-DARE-merge-v7.Q4_0.gguf	Q4_0	18.13GB
Yi-34B-200K-DARE-merge-v7.IQ4_NL.gguf	IQ4_NL	18.3GB
Yi-34B-200K-DARE-merge-v7.Q4_K_S.gguf	Q4_K_S	18.25GB
Yi-34B-200K-DARE-merge-v7.Q4_K.gguf	Q4_K	19.24GB
Yi-34B-200K-DARE-merge-v7.Q4_K_M.gguf	Q4_K_M	19.24GB
Yi-34B-200K-DARE-merge-v7.Q4_1.gguf	Q4_1	20.1GB
Yi-34B-200K-DARE-merge-v7.Q5_0.gguf	Q5_0	22.08GB
Yi-34B-200K-DARE-merge-v7.Q5_K_S.gguf	Q5_K_S	22.08GB
Yi-34B-200K-DARE-merge-v7.Q5_K.gguf	Q5_K	22.65GB
Yi-34B-200K-DARE-merge-v7.Q5_K_M.gguf	Q5_K_M	22.65GB
Yi-34B-200K-DARE-merge-v7.Q5_1.gguf	Q5_1	24.05GB
Yi-34B-200K-DARE-merge-v7.Q6_K.gguf	Q6_K	26.28GB
Yi-34B-200K-DARE-merge-v7.Q8_0.gguf	Q8_0	34.03GB

Original model description:

language:

en license: other library_name: transformers tags:
mergekit
merge
Yi license_name: yi-license license_link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE base_model: [] model-index:
name: Yi-34B-200K-DARE-merge-v7 results:
- task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
  - type: acc_norm value: 68.09 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
  - type: acc_norm value: 85.99 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
  - type: acc value: 77.3 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
  - type: mc2 value: 58.9 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
  - type: acc value: 83.11 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
  - type: acc value: 65.35 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=brucethemoose/Yi-34B-200K-DARE-merge-v7 name: Open LLM Leaderboard

Possibly made obsolete by: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megamerge-v8

Yi 34B 200K DARE Merge v7

A merge of several Yi 34B 200K models using the new DARE Ties method via mergekit. The goal is to create a merge model that excels at 32K+ context performance.

Prompt template: Orca-Vicuna

SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:

It might recognize ChatML, and possibly Alpaca-like formats. Raw prompting as described here is also effective: https://old.reddit.com/r/LocalLLaMA/comments/18zqy4s/the_secret_to_writing_quality_stories_with_llms/

Running

Being a Yi model, try running a lower temperature with 0.02-0.06 MinP, a little repetition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default, and it really needs a low temperature + MinP to cull the huge vocabulary.

24GB GPUs can efficiently run Yi-34B-200K models at 45K-90K context with exllamav2, and performant UIs like exui. I go into more detail in this post. 16GB GPUs can still run the high context with aggressive quantization.

To load/train this in full-context backends like transformers, you must change max_position_embeddings in config.json to a lower value than 200,000, otherwise you will OOM! I do not recommend running high context without context-efficient backends like exllamav2 or unsloth.

Testing Notes

See: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-merge-v5#testing-notes

A "4k" merge model was created to try and extend the context of SUS Chat and DPO-bagel before adding them to the merge: https://huggingface.co/brucethemoose/SUS-Bagel-200K-DARE-Test

In addition, the weight gradients are biased towards Vicuna-format models in the first few layers to try and "emphasize" the Orca-Vicuna prompt template. How sucessful this is remains to be seen.

Merge Method

This model was merged using the DARE TIES merge method using /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

yaml

models:
  - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
    # No parameters necessary for base model
  - model: /home/alpha/Storage/Models/Raw/migtissera_Tess-34B-v1.4
    parameters:
      weight: [0.23, 0.125, 0.125, 0.125, 0.125, 0.125]
      density: 0.59
  - model: /home/alpha/Models/Raw/Mihaiii_Pallas-0.5
    parameters:
      weight: [0.23, 0.125, 0.125, 0.125, 0.125, 0.125]
      density: 0.59
  - model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
    parameters:
      weight: [0.02, 0.106, 0.106, 0.106, 0.106, 0.106]
      density: 0.59
  - model: /home/alpha/Storage/Models/Raw/jondurbin_bagel-34b-v0.2
    #Only the SFT in the main merge since the DPO version seems to have no long context ability at all
    parameters:
      weight: [0.02, 0.100, 0.100, 0.100, 0.100, 0.100]
      density: 0.4
  - model: /home/alpha/Storage/Models/Raw/kyujinpy_PlatYi-34B-200k-Q-FastChat
    parameters:
      weight: [0.02, 0.100, 0.100, 0.100, 0.100, 0.100]
      density: 0.59
  #- model: /home/alpha/Storage/Models/Raw/ehartford_dolphin-2.2-yi-34b-200k
  #  Dolphin 200K seems to be funky according to multiple leaderboards and perplexity tests?
  #  parameters:
  #    weight: 0.15
  #    density: 0.6
  - model: /home/alpha/Models/Raw/adamo1139_Yi-34B-200K-AEZAKMI-v2
    parameters:
      weight: [0.02, 0.110, 0.110, 0.110, 0.110, 0.110]
      density: 0.59
  - model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
    parameters:
      weight:  [0.22, 0.126, 0.126, 0.126, 0.126, 0.126]
      density: 0.59
  - model: /home/alpha/Storage/Models/Raw/4kmerge
    parameters:
      weight: [0.02,  0.108, 0.108, 0.108, 0.108, 0.108]
      density: 0.5
  - model: /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
    parameters:
      weight: [0.22, 0.100, 0.100, 0.100, 0.100, 0.10]
      density: 0.59
merge_method: dare_ties
tokenizer_source: union
base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
parameters:
  int8_mask: true
dtype: bfloat16

The following config was used for the "4kmerge" model: