There are already many GGUF builds of this model, several of them excellent and far more
downloaded than this one will ever be. This repo exists for one reason: it publishes what the
others don't — how far each quant's output distribution actually drifts from the unquantized
model, measured with KL divergence and top-1 agreement, not asserted.
And that measurement turned up something worth your attention before you pick a file.
⚠️ Read this before choosing by perplexity
quant
size
PPL (wikitext-2)
ΔPPL vs BF16
KLD median
KLD p95
Top-1 agreement
Q3_K_M
13.50 GB
5.604
−0.360
0.0202
0.3262
89.32 %
IQ4_XS
15.31 GB
6.119
+0.156
0.0061
0.1023
93.29 %
Q4_K_M
16.81 GB
5.975
+0.012
0.0050
0.0761
94.29 %
Reference: BF16 GGUF, PPL 5.963.
Q3_K_M has the best perplexity of the three — better than the unquantized model itself. If this
card only reported PPL, like most quant repos do, the obvious read would be "take the smallest one,
it even beats the original."
That read is wrong. By KL divergence Q3_K_M is the least faithful of the three: its median
divergence is 4× Q4_K_M's, and it picks a different most-likely token than BF16 on 11 % of
positions versus 5.7 % for Q4_K_M. It isn't better — it's different, and wikitext happens to
reward that particular difference. Perplexity measures how comfortable a model is on one corpus;
it does not measure whether you still have the model you think you have.
That is the whole argument for KLD, and it is why this table exists.
📈 Charts
KLD vs PPL
Top-1 agreement
KLD distribution
The third chart is worth a look: divergence is not spread evenly. The median token is nearly
untouched at every tier — it's the tail that carries the damage, and the tail is where reasoning
chains and rare tokens live.
✅ Recommended files
you want
take
why
the closest thing to the original
Q4_K_M
best KLD and best top-1 agreement here
a good size/fidelity compromise
IQ4_XS
1.5 GB smaller than Q4_K_M, still 93.3 % top-1
the smallest that still works
Q3_K_M
genuinely lossier — see the warning above
📦 Files
file
size
bits
imatrix
Qwen3.6-27B-Q3_K_M.gguf
13.50 GB
~3
yes
Qwen3.6-27B-IQ4_XS.gguf
15.31 GB
~4
yes
Qwen3.6-27B-Q4_K_M.gguf
16.81 GB
~4
yes
Qwen3.6-27B.imatrix
13.6 MB
—
the importance matrix itself
Q5_K_M and Q6_K are not here. They were not built — the machine that produced this ladder ran
out of storage, and the tiers were built smallest-first on purpose so that the sizes most people
download survived the cut. Saying so is cheaper than pretending the ladder is complete.
🧠 What this model actually is (it matters for context)
Qwen3.6-27B is not a plain transformer. Of its 65 blocks:
48 are SSM / linear-attention layers (ssm_conv, ssm_scan, gated deltas),
16 are full-attention layers — every 4th block,
1 is an MTP (multi-token-prediction) head.
Only those 16 attention layers carry a KV cache that grows with sequence length. That is why a
262,144-token context is tractable here in a way it would not be for a dense 27B, and it is the
most interesting property of this model.
About the MTP head: it is present in these files, and llama.cpp explicitly ignores it
(unused tensor blk.64.nextn.* — ignoring). It costs you a little disk and does nothing under
llama.cpp today. Repos with -MTP- in the name are making the same tensors available for runtimes
that do use them.
-no-cnv matters: without it recent llama-cli builds drop into conversation mode and wait on
stdin, which looks exactly like a hang if you are scripting it.
Context Modelfiles for 8K / 32K / 128K / 262K are in configs/, with the memory
trade-off written out in CONTEXT.md.
🔬 Evaluation methodology
Reproducibility is the point of this repo, so here is exactly how every number above was produced.
Reference: BF16 GGUF converted from the original safetensors, 866 tensors, verified complete
by walking every tensor offset to the end of the file.
Eval corpus: wikitext-2 test split — deliberately disjoint from the imatrix
calibration corpus (source code + general prose). Measuring on your calibration data inflates
the result.
Sample size: 8 chunks × 2048 tokens = 16,384 tokens. This is small, and it is small for a
concrete reason: this model's vocabulary is 248,320 tokens, so llama.cpp's reference-logit file
costs ~497 KB per token. The direction of the Q3_K_M effect is solid
(mean ln(PPL(Q)/PPL(base)) = −0.062 ± 0.011), but treat the magnitudes as indicative, not tight.
Metrics: llama-perplexity --kl-divergence --kl-divergence-base, giving KLD mean/median/
percentiles, Δp statistics and top-1 agreement ("same top p"). Per-quant logs in
reports/.
imatrix: 277 chunks over a ~513 KB corpus, 992 entries covering blocks 0–63. Block 64 (the
MTP head) has no coverage because llama.cpp does not use it.
Quantization source: every tier was quantized from the BF16, never from another quant —
deriving a tier from a tier compounds error and would make this table measure the wrong thing.
Max KLD is ~27 for all three tiers. That is a handful of rare tokens where any quantization
falls apart; read the median and p95, not the max.
What is NOT measured here
No agentic/SWE benchmark was run. This repo makes no claim about tool use or coding ability.
The 262,144 context is the model's declared native window, not a verified one — a
needle-in-a-haystack run at 128K/256K is pending. Do not read "262K" here as "tested at 262K".
Other GGUF builds of this model, offered honestly because several are better maintained and more
complete than this one: unsloth,
bartowski,
lmstudio-community,
ggml-org. If you want the full ladder or the MTP
variants, go there. Come here for the numbers.
🗒️ Changelog
2026-08-14 — first release: Q3_K_M, IQ4_XS, Q4_K_M with imatrix, KLD/PPL/top-1 table, charts,
scripts and checksums. Q5_K_M/Q6_K deferred (storage). Needle test and agentic probe pending.
