GGUF quantizations of barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP — a REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE, merged with an Opus-reasoning-distilled LoRA adapter, and grafted with a Multi-Token Prediction (MTP) head reverse-engineered from the original unpruned checkpoint.
Converted with llama.cpp's convert_hf_to_gguf.py. The MTP head is preserved as native nextn.* / blk.40.* tensors — no manual tensor mapping was required.
+39% to +67% faster generation via self-speculative decoding — measured with llama-bench/llama-cli, 5 runs per config, not estimated. See Benchmarks.
MTP costs zero measured quality — identical scores to four decimal places vs. the non-MTP release. The speedup above is free. See Quality benchmarks.
The reasoning LoRA beats the unpruned base model on ARC-Challenge (0.616 vs 0.532) — a net skill gain from distillation, not just recovered pruning loss.
Files
Quant
Size
Notes
BF16
57.9 GB
full precision reference
Q8_0
30.8 GB
near-lossless
Q6_K
23.8 GB
Q5_K_M
20.7 GB
Q4_K_M
17.7 GB
most popular K-quant
Q4_K_S
16.6 GB
IQ4_XS
15.7 GB
imatrix-based, smaller & often better than Q4_K_S
Q3_K_M
14.0 GB
IQ3_M
12.9 GB
IQ3_XXS
11.5 GB
Q2_K
10.9 GB
IQ2_M
10.0 GB
IQ2_XXS
8.2 GB
IQ1_M
7.2 GB
extreme, quality loss is significant
All quants ≤Q4_K_M were calibrated with an importance matrix (imatrix) built from 512 samples of barozp/opus-reasoning-distill-train. K-quants (Q*_K*) favor broad compatibility and fast CPU inference; IQ-quants (IQ*) require the imatrix and give better quality per bit at ≤4-bit, at some CPU-inference speed cost.
Using the MTP head (speculative decoding)
The GGUF ships the MTP decoder layer as nextn.* (front projection) and blk.40.* (the layer itself). Modern llama.cpp builds can use it for self-speculative decoding — the MTP head drafts a token, the base model verifies it in the same pass.
This is not automatic on every build. Enable it explicitly:
Without --spec-type draft-mtp (or your frontend's equivalent toggle), generation uses only the base model — behavior and speed then match the non-MTP release.
Recommended draft settings: max draft tokens 3, min draft tokens 0, draft probability 0
(llama.cpp CLI: --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0). Some frontends
ship stricter defaults (e.g. LM Studio defaults to draft probability 0.75), which noticeably
suppresses acceptance — lowering it to 0 closes most of the gap to the measured rate below.
Measured MTP-head acceptance rate on held-out validation data: ~0.695 (69.5%).
Benchmarks
Measured with llama-cli (Q4_K_M, flash attention on, greedy decoding, 5 runs per config, mean ± std) on an NVIDIA RTX PRO 6000 Blackwell Server Edition (97 GB VRAM).
-ngl
Config
tok/s (generation)
Speedup vs. no-MTP baseline
99 (full offload)
No-MTP baseline
219.06 ± 0.10
—
99 (full offload)
MTP, spec decoding off
219.34 ± 0.10
+0.1% (noise)
99 (full offload)
MTP, spec decoding on
304.56 ± 0.36
+39.0%
20 (partial offload)
No-MTP baseline
58.60 ± 0.52
—
20 (partial offload)
MTP, spec decoding off
56.80 ± 0.78
−3.1% (noise)
20 (partial offload)
MTP, spec decoding on
97.60 ± 0.78
+66.6%
Takeaways:
With speculative decoding off, the MTP file performs identically to the non-MTP file — the extra head carries no measurable VRAM/compute penalty when idle.
With speculative decoding on, the MTP head gives a substantial, reproducible speedup (39–67% in this environment) — larger at partial GPU offload than at full offload.
Speculative decoding's benefit depends on whether generation is compute-bound or memory-bandwidth-bound. On systems where the bottleneck is CPU↔GPU memory bandwidth (e.g. heavy CPU offload on a smaller/weaker GPU), the draft+verify overhead can compete with an already-scarce resource and speculative decoding may show little gain or even a small regression. If you observe a slowdown with spec decoding on, disable it (--spec-type none) or use the non-MTP release — both give identical baseline speed.
Quality benchmarks
Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled — see note) on the underlying safetensors checkpoint, against the same base chain: unpruned Qwen3.6-35B-A3B → REAP 205/256 pruning only (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) → the non-MTP merge → this checkpoint.
Task
Axis
Qwen3.6-35B-A3B
RangerX REAP-20%
Ours, no MTP
This model (+MTP)
wikitext perplexity (lower=better)
sanity
7.85
10.06
10.0635
10.0635
MMLU
out-of-calibration
0.8409
0.8152
0.8257
0.8257
HellaSwag (0-shot, acc_norm)
out-of-calibration
0.7420
0.7440
0.7340
0.7340
ARC-Challenge (0-shot, acc_norm)
out-of-calibration
0.5320
0.5340
0.6160
0.6160
MTP head has zero measured effect on these scores — this checkpoint and the non-MTP release score identically to four decimal places on every task, exactly as expected since transformers never loads the MTP tensors through the standard loading path (see note above). Use this release for the self-speculative-decoding speedup (benchmarks above), not for a quality difference — quality-wise the two releases are interchangeable.
Separately, the reasoning LoRA (present in both releases) pushes ARC-Challenge above the unpruned 256-expert base model (0.616 vs 0.532) — a genuine reasoning-skill transfer from the Opus chain-of-thought training data. MMLU sees a smaller but real gain (+1.3pp retained vs. the LoRA-less pruned checkpoint); wikitext perplexity is unaffected; HellaSwag is flat within measurement noise.
Note: chat template was tested and found to badly corrupt loglikelihood-based multiple-choice scoring for this model family (MMLU dropped from 0.85 to 0.38 on the base model with it on) — all numbers above are with it off, applied consistently across every checkpoint. HellaSwag/ARC-Challenge are 0-shot, also applied consistently. Full methodology on the safetensors model card.
Credits
Base architecture: Qwen3.6-35B-A3B (MoE), pruned via REAP (Router-weighted Expert Activation Pruning) to 205/256 experts.
