You can deploy this model and start earning money today!

RichardErkhov/
mlabonne_-_BigQwen2.5-Echo-47B-Instruct-gguf

Name: mlabonne_-_BigQwen2.5-Echo-47B-Instruct-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

BigQwen2.5-Echo-47B-Instruct - GGUF

Model creator: https://huggingface.co/mlabonne/
Original model: https://huggingface.co/mlabonne/BigQwen2.5-Echo-47B-Instruct/

Name	Quant method	Size
BigQwen2.5-Echo-47B-Instruct.Q2_K.gguf	Q2_K	16.45GB
BigQwen2.5-Echo-47B-Instruct.IQ3_XS.gguf	IQ3_XS	18.31GB
BigQwen2.5-Echo-47B-Instruct.IQ3_S.gguf	IQ3_S	19.32GB
BigQwen2.5-Echo-47B-Instruct.Q3_K_S.gguf	Q3_K_S	19.26GB
BigQwen2.5-Echo-47B-Instruct.IQ3_M.gguf	IQ3_M	19.82GB
BigQwen2.5-Echo-47B-Instruct.Q3_K.gguf	Q3_K	21.35GB
BigQwen2.5-Echo-47B-Instruct.Q3_K_M.gguf	Q3_K_M	21.35GB
BigQwen2.5-Echo-47B-Instruct.Q3_K_L.gguf	Q3_K_L	23.16GB
BigQwen2.5-Echo-47B-Instruct.IQ4_XS.gguf	IQ4_XS	23.97GB
BigQwen2.5-Echo-47B-Instruct.Q4_0.gguf	Q4_0	25.02GB
BigQwen2.5-Echo-47B-Instruct.IQ4_NL.gguf	IQ4_NL	25.26GB
BigQwen2.5-Echo-47B-Instruct.Q4_K_S.gguf	Q4_K_S	25.21GB
BigQwen2.5-Echo-47B-Instruct.Q4_K.gguf	Q4_K	26.65GB
BigQwen2.5-Echo-47B-Instruct.Q4_K_M.gguf	Q4_K_M	26.65GB
BigQwen2.5-Echo-47B-Instruct.Q4_1.gguf	Q4_1	27.74GB
BigQwen2.5-Echo-47B-Instruct.Q5_0.gguf	Q5_0	30.45GB
BigQwen2.5-Echo-47B-Instruct.Q5_K_S.gguf	Q5_K_S	30.45GB
BigQwen2.5-Echo-47B-Instruct.Q5_K.gguf	Q5_K	31.29GB
BigQwen2.5-Echo-47B-Instruct.Q5_K_M.gguf	Q5_K_M	31.29GB
BigQwen2.5-Echo-47B-Instruct.Q5_1.gguf	Q5_1	33.16GB
BigQwen2.5-Echo-47B-Instruct.Q6_K.gguf	Q6_K	36.22GB
BigQwen2.5-Echo-47B-Instruct.Q8_0.gguf	Q8_0	46.91GB

Original model description:

language:

en license: apache-2.0 library_name: transformers tags:
mergekit
merge
lazymergekit base_model:
Qwen/Qwen2.5-32B-Instruct license_name: tongyi-qianwen license_link: https://huggingface.co/Qwen/Qwen2-72B-Instruct/blob/main/LICENSE pipeline_tag: text-generation model-index:
name: BigQwen2.5-Echo-47B-Instruct results:
- task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: num_few_shot: 0 metrics:
  - type: inst_level_strict_acc and prompt_level_strict_acc value: 73.57 name: strict accuracy source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: num_few_shot: 3 metrics:
  - type: acc_norm value: 44.52 name: normalized accuracy source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competition_math args: num_few_shot: 4 metrics:
  - type: exact_match value: 3.47 name: exact match source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: num_few_shot: 0 metrics:
  - type: acc_norm value: 8.61 name: acc_norm source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: num_few_shot: 0 metrics:
  - type: acc_norm value: 10.19 name: acc_norm source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: num_few_shot: 5 metrics:
  - type: acc value: 41.49 name: accuracy source: url: >- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct name: Open LLM Leaderboard

BigQwen2.5-Echo-47B-Instruct

BigQwen2.5-Echo-47B-Instruct is a Qwen/Qwen2-32B-Instruct self-merge made with MergeKit.

🔉 Echo Merge

I've tried a more gradual approach with a distributed repetition pattern. Instead of replicating blocks of 8 or more layers, I'm replicating individual layers in these blocks:

First 8 layers: No replication
Next 8 layers: Replicate 2 layers (first one, middle one)
Next 8 layers: Replicate 4 layers (1st, 3rd, 5th, 7th)
Next 8 layers: Replicate 8 layers (all of them)
Next 8 layers: Replicate 4 layers (1st, 3rd, 5th, 7th)
Next 8 layers: Replicate 2 layers (first one, middle one)
First 8 layers: No replication

I used this string to visualize it, where 0 are original layers and 1 duplicated ones (the order doesn't matter):

00000000 1000010000 100100100100 1010101010101010 1010101010101010 100100100100 1000010000 00000000

The main idea is that the input/output difference of middle layers is quite small, so replicating a middle layer has a small impact on the output. The additional layers are designed to increase the model's capacity without breaking the information flow, which often creates "insane" self-merges.

🏆 Evaluation

Metric	BigQwen2.5-Echo-47B-Instruct	BigQwen2.5-52B-Instruct	Qwen2.5-32B-Instruct
Avg.	30.31	37.42	36.17
IFEval (0-Shot)	73.57	79.29	83.46
BBH (3-Shot)	44.52	59.81	56.49
MATH Lvl 5 (4-Shot)	3.47	17.82	0
GPQA (0-shot)	8.61	6.94	11.74
MuSR (0-shot)	10.19	10.45	13.5
MMLU-PRO (5-shot)	41.49	50.22	51.85

🧩 Configuration

The following YAML configuration was used to produce this model:

yaml

slices:
  # First 8 layers: No replication
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [0, 8]

  # Next 8 layers: Replicate 2 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [8, 9]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [8, 9]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [9, 13]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [13, 14]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [13, 14]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [14, 16]

  # Next 8 layers: Replicate 4 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [16, 18]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [17, 19]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [18, 20]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [19, 21]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [20, 22]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [21, 23]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [22, 24]

  # Next 8 layers: Replicate all 8 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [24, 25]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [24, 26]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [25, 27]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [26, 28]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [27, 29]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [28, 30]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [29, 31]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [30, 32]

  # Middle 8 layers: Replicate all 8 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [32, 33]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [32, 34]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [33, 35]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [34, 36]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [35, 37]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [36, 38]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [37, 39]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [38, 40]

  # Next 8 layers: Replicate 4 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [40, 42]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [41, 43]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [42, 44]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [43, 45]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [44, 46]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [45, 47]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [46, 48]

  # Next 8 layers: Replicate 2 layers
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [48, 49]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [48, 49]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [49, 53]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [53, 54]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [53, 54]
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [54, 56]

  # Last 8 layers: No replication
  - sources:
    - model: Qwen/Qwen2.5-32B-Instruct
      layer_range: [56, 64]

merge_method: passthrough
dtype: bfloat16

💻 Usage

python

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "mlabonne/BigQwen2.5-Echo-47B-Instruct"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.

RichardErkhov/mlabonne_-_BigQwen2.5-Echo-47B-Instruct-gguf