UD-IQ1_M and UD-IQ2_S were briefly published broken, then withdrawn, and are now
rebuilt, load-tested, and back. If you downloaded either before 2026-08-21 22:00 UTC,
re-download. No other tier was ever affected.
If you are quantizing this model yourself, this is the part worth reading. Upstream
builds low-bit tiers with a different architecture, not just different bit widths:
Q3_K_XL and above
IQ1_M / IQ2_S
tensors
506
498
highest block
64 (MTP head at q6_k)
63 (MTP pruned)
block_count
65
64
nextn_predict_layers
1
0
Their imatrix has 496 entries because it matches that pruned topology. Quantizing a
65-block f16 against it fails with
Missing importance matrix for tensor blk.64.attn_k.weight.
--prune-layers 64 alone is not enough — it removes the tensors but leaves the model
declaring an MTP head, so loading dies on blk.63.nextn.eh_proj.weight not found: a
structurally valid file containing an unloadable model. You also have to rewrite the
metadata:
Every file here is now gated on exit code → magic bytes → an actual load test before
upload. Header checks alone let two broken builds through; only loading the model catches it.
Uncensored Qwen3.8-27B, quantized with per-tensor dynamic bit allocation.
Most uncensored GGUFs are uniform quants — every layer gets the same treatment. These use a
per-tensor type map, so sensitive tensors keep precision while tolerant ones compress hard.
The result: a Q3 that outperforms a Q4 3.3 GB larger, with full 250k-token retrieval.
Provenance, stated plainly. The per-tensor quant map was extracted from
unsloth/Qwen3.8-27B-GGUF
(Unsloth Dynamic 3.0) by reading the
tensor-type table from their published files, then applied to
JonathanColetti/Qwen3.8-27B-Uncensored
weights with that repo's imatrix. The recipe is Unsloth's work, not ours — we did not
independently derive it. All upstream projects are Apache 2.0. See Credits.
Measurements
Every number below is from one machine, one harness, same session — RTX 4090 24 GB,
llama.cpp with DFlash2 speculative decoding, q4_0 KV cache, -c 262144,
wikitext-2 perplexity at 60 chunks.
⚠️ Do not compare these PPL values against numbers from other setups. Chunk count and
harness change the absolute scale; only within-table comparisons are meaningful.
Quant
Size
PPL ↓
Median tok/s
Needle retrieval
Uncensored
Unleashed UD-IQ4_XS
14.3 GB
6.3502
—
(pending)
✅
Unleashed UD-Q3_K_XL
13.2 GB
6.4036
112.8
✅ 250,806 tok
✅
Unleashed UD-IQ3_XXS
11.0 GB
6.4818
—
(pending)
✅
unsloth UD-Q3_K_XL (ref)
12.24 GB
6.3993
110.7
✅ 250,806 tok
❌
uncensored IQ4_XS (ref)
15.1 GB
6.4149
107.3
32k
✅
unsloth UD-Q4_K_XL (ref)
16.7 GB
6.4181
62.6
4k
❌
uniform Q3_K_M + imatrix (ref)
12.57 GB
6.5316
84.8
✅ 258,900 tok
✅
unsloth UD-Q2_K_XL (ref)
9.15 GB
6.6469
—
—
❌
Tiers marked (pending) are uploaded and usable; their full speed/retrieval sweep is still running.
What the table says:
The dynamic Q3 beats every Q4 tested, including a Q4 2.2 GB larger.
Against uniform Q3 on identical weights: −0.13 PPL and +33% throughput for 0.6 GB more.
Quality matches Unsloth's own Q3 within error (Δ 0.0043, σ ≈ 0.045) — call it a tie,
with the difference being uncensored weights at ~1 GB more.
Q2 is the floor. The dynamic recipe halves the damage of a naive Q2 but still costs
~3.8%. Q3 is where quality holds.
Speed: what you should actually expect
Measured on real traffic (n=1,696 requests, RTX 4090, Q3_K_XL, dflash2 draft). Throughput
depends far more on how many tokens you generate than on context depth, because every
request pays fixed overhead (prefill, sampler init, draft warm-up) that only amortises over
a long generation:
Tokens generated
n
Median tok/s
1-50 (tool calls, short acks)
1,289
24.9
51-200
97
32.6
201-600
191
43.6
601-1500
61
46.9
1500+ (long prose, code)
58
56.3
Peak observed on a single request: ~195 tok/s. Do not plan around peak.
If you are running an agent that mostly emits short tool calls, expect the ~25 tok/s end of
that table, not the top. That is not a defect of this quant - it is the fixed cost of a
27B model per request, and it applies to any GGUF of this model. Speculative decoding needs
a few tokens to build acceptance momentum, so very short replies finish before the draft
warms up.
Lowering context does not fix this. Only 17 of 65 layers are full-attention (the rest are
linear/DeltaNet), so KV work is a small share of per-token cost. We measured 61 tok/s median at
16-64k context versus 30 tok/s at <4k - deeper context was faster, because those requests
generated more tokens. Context depth is not the lever; generation length is.
Retrieval at depth
Exact-string needle recovered from a 250,806-token context (98% window occupancy):
Context depth
Prompt tokens
Result
Gen tok/s
32k
31,265
✅ found
50.4
120k
119,779
✅ found
60.6
250k
250,806
✅ found
40.4
Files
All nine tiers, same recipe, same weights, plus a tested vision projector.
Sizes are on-disk (decimal GB).
File
Size
Fits (weights + 262k ctx)
Notes
UD-IQ1_M
6.8 GB
8 GB card
⚠️ scores at chance (25.8% MMLU) — published for completeness, not for use
VRAM math for a 24 GB card: weights + ~5 GB KV (262k @ q4_0) + ~2 GB draft model +
compute buffers. The KV cache is small for a 27B because Qwen3.8 is hybrid DeltaNet —
only 17 of 65 layers are full attention.
External context for the base model (stock Qwen3.8-27B, bf16 — not our quants):
Artificial Analysis (Aug 2026): ranks alongside DeepSeek V4 and GPT-5.6-class models on the
Intelligence Index — discussion ·
charts
"Beats all medium models (40B–150B); same score as DeepSeek V4 Flash" — HN summary of AA data
Hybrid DeltaNet architecture: only 17 of 65 layers are full attention → KV cache at 262k ctx is
~5 GB (q4_0), which is why full context fits on a 24 GB card at all
Our quants inherit these capabilities minus quantization loss (measured above).
Optimal settings — these matter
Setting
Value
Why
temperature
1.0
Qwen3.8 published default; pairs with top_p 0.95 / top_k 20
top_p / top_k / min_p
0.95 / 20 / 0.0
stock Qwen3.8 recommendation
repeat_penalty
1.0
K-quants at this size don't loop; leave it off
KV cache
q4_0
required for 262k ctx on 24 GB; ≤131k ctx can use q8_0
reasoning effort
low default, raise per-task
thinking burns output-token budget fast
enable_thinking
off for code-heavy tasks
reasoning chains eat the output budget
Which tier should you actually use?
Every tier was measured on the same box (RTX 4090, llama.cpp-dflash2) on the same day:
wiki.test perplexity, generation speed, MMLU (10 questions x 57 subjects = 570, greedy,
thinking disabled), needle-in-haystack retrieval, and a 20-prompt refusal battery.
Tier
Size
PPL ↓
tok/s
MMLU ↑
Needle retrieval
Refusal
UD-IQ1_M
6.8 GB
8.4493
77.0
25.83% ⚠️
❌ failed @32k
0.0%
UD-IQ2_S
8.5 GB
7.0195
66.6
70.83%
✅
0.0%
UD-Q2_K_XL
9.9 GB
6.6367
68.2
70.83%
✅
0.0%
UD-IQ3_XXS
11.0 GB
6.4818
62.0
75.00%
✅
0.0%
UD-Q3_K_XL
13.2 GB
6.4036
52.2
82.50%
✅
0.0%
UD-IQ4_XS
14.3 GB
6.3502
49.5
81.67%
✅
0.0%
UD-Q4_K_M
16.5 GB
6.3857
48.0
78.33%
✅
0.0%
UD-Q5_K_M
19.8 GB
6.4395
41.1
79.17%
✅
0.0%
UD-Q6_K
22.1 GB
6.4065
37.8
79.17%
✅
0.0%
MMLU here is n=120 per tier (subject-stratified) for ladder comparison; the headline 82.98%
figure below is the full n=570 run on Q3_K_XL. n=120 carries roughly ±4pp, so treat small
gaps as noise.
Three findings worth stating plainly
0. UD-IQ1_M scores at random chance. Do not use it. MMLU is 4-way multiple choice, so
25% is the floor for guessing. IQ1_M scored 25.83% — statistically indistinguishable from
noise — with perplexity 32% worse than Q3 and a failed needle retrieval at 32k. It is the
fastest file here (77 tok/s) and it retains no usable knowledge. We are shipping it with the
numbers rather than quietly omitting the tier: if you have an 8 GB card, run UD-IQ2_S
instead — it is 1.7 GB larger, holds 70.83% MMLU, and passes retrieval.
The degradation is not gradual. IQ2_S matches Q2_K_XL exactly (70.83%) while being 1.4 GB
smaller; one step further down falls off a cliff. The usable floor of this model is 2-bit.
1. Perplexity is not capability.UD-IQ4_XS has the best perplexity in the entire ladder
(6.3502) and is not the best model — it scores below UD-Q3_K_XL on MMLU while being
1.1 GB larger and 5% slower. If you are choosing a quant, choose on task benchmarks, not PPL.
2. Nothing above Q3 buys anything.UD-Q6_K is 8.9 GB larger and 28% slower than
UD-Q3_K_XL, and does not score better. Quality rises steeply from Q2 to Q3 and then flattens.
The recommended tier is the knee of every curve, not a compromise.
3. Abliteration survives quantization. Refusal rate is 0.0% at every tier, down to
9.9 GB. Whatever the low-bit tiers cost you, it is not censorship creeping back in.
Capability: MMLU
Measured on this exact quant (UD-Q3_K_XL), not the parent bf16 — so it includes
quantization loss and abliteration loss.
Model
MMLU (0-shot, 10/subject, n=570)
Unleashed UD-Q3_K_XL (13.2 GB, uncensored)
82.98% (473/570)
stock Qwen3.8-27B bf16, same protocol (published by OBLITERATUS)
85.3%
Protocol: greedy (temperature 0), thinking disabled, single-letter answer extraction,
10 questions per subject across all 57 MMLU subjects, 0 parse failures.
Reading this honestly: ~2.3 points below stock bf16, and that gap covers both the
abliteration and 3-bit quantization. For a 13.2 GB file that also keeps 262k context and
runs at 25-56 tok/s on one consumer GPU (see the speed section - it depends heavily on
how many tokens you generate), that is the trade being offered. If you need every
last point of benchmark accuracy and don't need uncensored, run the stock bf16.
Refusal behaviour
Model
Refusal rate
Unleashed UD-Q3_K_XL
0 / 20 (0.0%)
stock Qwen3.8-27B
refuses most of this set
20-prompt boundary battery (security mechanics, drug pharmacology, graphic fiction, social
engineering, morally repugnant character writing, suggestive romance). Scored by refusal-marker
detection on the response opening; every prompt produced a substantive answer (806-1004 chars
on the last three).
Does the abliteration method matter?
We tested this directly. Taking a different uncensored base — OBLITERATUS
V2, which uses complementary
blending (SVD + LEACE) rather than the Heretic method — and running it through the identical
UD-Q3_K_XL recipe, same hardware, same day, so the only variable was the base weights:
Base
MMLU (n=570)
Refusal
Heretic-abliterated (this repo)
82.98% (473/570)
0/20
SVD+LEACE-abliterated, same recipe
82.98% (473/570)
0/20
Identical scores. The two models disagree on 44 individual questions (92.3% prediction
agreement) — they are genuinely different models that happen to land in the same place.
At 3-bit, the abliteration technique appears to be roughly fungible: different methods, same
measured capability, same refusal behaviour. Worth knowing before you switch bases chasing a
benchmark claim.
Caveat, stated plainly: n=20 is a spot check, not a 800-prompt corpus. The parent model's
card describes refusals as substantially reduced, not eliminated - you will likely find edge
cases. This measures our quant of it, at n=20, on the prompts listed in refusal-results.jsonl.
MTP and vision
MTP head: present. The nextn.* multi-token-prediction tensors survived abliteration
(re-grafted by the parent repo) and are retained in every quant here (eh_proj at Q6_K).
Vision: bundled and tested.mmproj-Unleashed-f16.gguf (0.93 GB) is in this repo,
built from the same parent weights as these quants. Every other uncensored GGUF of this
model we are aware of is text-only, because convert_hf_to_gguf.py defaults to text and
drops the 333 vision tensors unless you pass --mmproj.
bash
1# vision is one extra flag - the weights themselves are unchanged2llama-server \3 -m Qwen3.8-27B-Unleashed-UD-Q3_K_XL.gguf \4 --mmproj mmproj-Unleashed-f16.gguf \5 -ngl 999 -c 262144 -fa on \6 --cache-type-k q4_0 --cache-type-v q4_0 --jinja
Then post images to /v1/chat/completions with an image_url content part (OpenAI format),
or use llama-mtmd-cli.
What we verified (Q3_K_XL + this projector, single 4090):
Check
Result
loads alongside a 3-bit quant
pass - 15.9 GB VRAM at 32k ctx
shape + colour recognition
pass - "a solid red circle"
spatial reasoning (left/right)
pass - "LEFT is blue, RIGHT is yellow"
Tested with synthetic images generated at test time, so ground truth is exact and the
model cannot be scored generously. Abliteration did not break cross-modal alignment -
that was the open risk and it came back clean.
Cost: the projector adds ~0.9 GB resident VRAM. Text-only requests never touch the vision
encoder, so there is no generation-speed penalty for having it loaded; you pay a one-time
prefill per image. Omit --mmproj and behaviour is identical to a text-only build.
Not yet measured by us: image-heavy benchmark accuracy, OCR quality, multi-image prompts.
Running it
Recommended llama.cpp server config — this is the setup all measurements were taken with:
Speculative decoding with DFlash2 (optional, big win)
Every speed number on this card was measured with a DFlash2 draft model attached. It is not
part of these files - it is a separate ~2 GB model that predicts several tokens ahead, which the
main model then verifies in one pass. On our runs it accepted 48.6% of drafted tokens with a
mean accepted run of 3.92 tokens. Note that acceptance on real agent traffic is lower
(median ~0.37-0.51) than on benchmark prompts.
Get the draft model from
spiritbuun/Qwen3.8-27B-DFlash2 (Q8_0, ~2 GB) and add:
Tuning note: we A/B'd the draft parameters. n_max=6, p_min=0.75 beat both n_max=10, p_min=0.55
(41% acceptance) and n_max=10, p_min=0.40 (30% acceptance). Higher p_min wins here - drafting
more aggressively costs more than it gains. Without a draft model everything still works, just
slower; subtract roughly a third of the throughput.
Notes:
--cache-reuse 256 matters more than it looks. With a stable system prompt, throughput at
120k depth measured higher than at 32k — prefix reuse amortizes the prefill.
q4_0 KV cache is what makes 262k fit in 24 GB. At f16 KV you will OOM well before that.
Sampler values are Qwen3.8's published recommendations, not tuned by us.
Method
Read the tensor-type table from each published Unsloth UD GGUF header (HTTP range request —
metadata only, no full download).
Emit a --tensor-type-file map, one tensor=type line per quantized tensor (506 for Q3_K_XL).
Run llama-quantize against jonathan's uncensored f16 with his published imatrix and that map.
The map is non-obvious and worth looking at. Across every tier from IQ1_S to Q8_K_XL, the
~96 ssm_alpha / ssm_beta tensors are held at Q8_0 — even in the 1-bit builds. Those gate
Qwen3.8's DeltaNet path. Our own hand-rolled attempt (uniform Q3_K on those tensors, heavier
attention) scored 6.5865 — the worst result we measured. The specific per-layer allocation
is the contribution, and it is Unsloth's.
Limitations
Uncensored ≠ unaligned. Refusal behaviour is substantially reduced, not eliminated.
See jonathan's card for measured refusal rates.
PPL is a weak proxy for task quality. A 16-task BenchLoop coding
suite (local-first LLM benchmark harness) scored identically (14/16) across every quant tested
here — it did not discriminate at this tier. Treat PPL as a relative signal.
Retrieval was tested with exact-string needles, which is the easy case. Multi-hop reasoning at
250k was not measured.
Single-machine numbers. No cross-hardware validation.
Vision: use the mmproj from the upstream repo; not included here.
Credits
JonathanColetti — the uncensored base weights
(abliteration via Heretic, MTP head re-grafted and verified)
and the imatrix used for these quants. None of this exists without that work.
Unsloth — the Dynamic 3.0 per-tensor
quantization recipe. We extracted and reapplied their map; the method and the analysis behind
it are entirely theirs. If you can use the censored base, use their files directly.
Apache 2.0, inherited from all upstream components. Unsloth and Qwen names are used descriptively
for attribution only; no affiliation or endorsement implied.
