GGUF quantizations of
barozp/Qwen3.8-27B-Opus-Distill —
a Qwen3.8-27B fine-tuned with LoRA on Claude Opus reasoning traces (merged),
with the native vision tower and native MTP head carried over untouched.
Highlights
Reasoning-distilled, not just quantized. The LoRA was trained on 14,250
Opus chain-of-thought traces and merged into the base weights. Quantization
only converts the weights — the reasoning gains travel with them unchanged.
Full multimodal. Native vision tower ships as a separate mmproj file
(~0.9 GB). Text-only users can ignore it entirely.
Native MTP for self-speculative decoding. The model was released with its
MTP head trained in — unlike grafted MTP setups, no approximation involved.
Free speedups on compute-bound hardware.
imatrix-calibrated. All quants below Q3_K_M use an importance matrix
built from the model's own reasoning-distillation data (see Imatrix).
Known issues
Reasoning loop under stacked output-format constraints. Reported by
zxbc2023 (full writeup, discussion #1).
Combining "no prose" with a second output-format constraint (e.g. "no markdown" or "no comments") can send this model into a non-converging
self-verification reasoning loop -- it burns the entire token budget with
zero visible output. Fully deterministic and reproducible at temp=0.
Root cause: traced to part of the training data being sourced from
reconstructed (not verbatim) Opus reasoning traces, not a capability gap.
Fixed in barozp/Qwen3.8-27B-Opus-Distill-v2
-- retrained on a rebuilt dataset where every row is traced to a verified
genuine source. If you're hitting this, switch to v2.
Workaround if staying on this version: avoid combining "no prose" with
another format constraint, or raise the generation token budget to >=4096
for constrained code-gen tasks.
Quality benchmarks (of the source safetensors model)
Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice),
chat template OFF, QUICK mode (--limit 500). Base and distill ran with the
identical harness, so the Δ column is the meaningful signal.
Task
Metric
Base
Distill
Δ
wikitext
word perplexity ↓
8.434
8.344
−0.09
mmlu
acc
0.849
0.849
−0.001
hellaswag
acc_norm
0.742
0.740
−0.002
arc_challenge
acc_norm
0.588
0.630
+0.042
gpqa_diamond
acc_norm
0.232
0.495
+0.263
Reading the table:
Reasoning improved (ARC +4.2pt, GPQA +26pt), knowledge stayed flat
(MMLU −0.001) and language modeling stayed flat (wikitext −0.09 ppl).
GPQA caveat: measured with thinking disabled (loglikelihood) — the base
scores near random (25%) because it gets no chance to deliberate. The +26pt Δ
is a valid same-protocol comparison, but do not compare 0.495 to Qwen's
published 89.2 (measured with thinking ON, different harness).
ARC-Challenge is saturated for modern models; treat it as continuity with
the Qwen3.6 release — GPQA is the stronger reasoning signal here.
Speed (MTP self-speculative decoding)
Not yet benchmarked for this exact model. On the Qwen3.6 sibling (same MTP
mechanism, grafted there), measured with llama.cpp: +39% tok/s full offload,
+67% partial offload with spec-decode ON. Native MTP (this model) is trained
in and typically does at least as well. Guidance:
K-quants (Q8_0–Q3_K_M) are plain llama-quantize passes, no imatrix needed.
IQ-quants (IQ3_XXS and below) require an importance matrix to run at all
in current llama.cpp and are built from the one in this repo (see below).
Which one to pick:
Best quality with headroom → Q6_K or Q8_0
Best quality/size balance → Q4_K_M (default recommendation)
imatrix.dat in this repo (512 samples from
barozp/opus-reasoning-distill-train,
context 512) was used to build the IQ quants above. It applies to any GGUF with
this same architecture — including the base
Qwen/Qwen3.8-27B — so it can be reused
for re-quantization without recomputing it:
Note on IQ1_M: the MTP head (blk.64, the nextn.* decoder layer) is
never exercised by a normal forward pass, so the imatrix has no data for it.
llama-quantize pins that block to q4_K instead of failing, which is why
IQ1_M lands at ~2.3 bits/weight (7.9 GB) rather than the ~1.8 a "pure"
IQ1_M would suggest — the MTP head alone accounts for the difference, the rest
of the model is quantized normally.
Vision (mmproj)
The vision tower is in Qwen3.8-27B-Opus-Distill-mmproj-f16.gguf (~0.9 GB) in
this repo. Load it alongside any quant for image/video input:
Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained
Notes
Thinking mode is on by default (same as the base model). The GGUF embeds
the chat template; how thinking is toggled depends on the llama.cpp version /
frontend (e.g., LM Studio exposes the setting in its UI).
Conversion: llama.cpp convert_hf_to_gguf.py from the corrected multimodal
config (nested text_config + vision_config).
No chaining: every quant was produced directly from the BF16 GGUF, so
errors do not accumulate across the ladder.
