Part of the Qwen3.8-27B mixed-precision EXL3 (measured) collection.

Four EXL3/Trellis quants of Qwen/Qwen3.8-27B, each optimizing a different constraint. Every quality claim is measured against a BF16 teacher via KL-divergence — no unverified accuracy assertions.

Model	Download	Resident	v5 mean KLD	Context (32 GB)	TL;DR
K5K6-hydrated	21.61 GB	20.31 GiB	0.002760	~180k	Best fidelity, smallest download, fastest cold start
K5K6	30.60 GB	20.32 GiB	0.003210	~180k	Attention width is a launch-time knob (K6/K5/K4)
K5K6-context	20.70 GB	18.41 GiB	0.003509	262,144	Native 262k context, int8 embeddings, hardware-qualified on RTX 5090
K4	28.31 GB	17.89 GiB	0.010604	262,144	Smallest footprint, native context, no overlay needed

All four are VLM (image-text-to-text) and require the Gilded Gnosis vLLM runtime — they do not load in stock vLLM, SGLang, or transformers.

Qwen3.8-27B EXL3 K5/K6 — lower divergence than official FP8 at 71 % of its memory

Requires a custom runtime. This checkpoint does not load in upstream vLLM, SGLang, TensorRT-LLM, llama.cpp, transformers, or stock exllamav3. It needs the Gilded Gnosis vLLM fork (public image below), explicit --quantization exl3, an exact ignore list, and — for CUDA graphs — an as-yet-unmerged patch (PR #314). Treat it as an experimental, runtime-specific research artifact.

QFS size–fidelity evidence

Which of the four builds

Measured results

Fidelity

Suite identity

Cumulative means at 10,480,640 positions

Paired per-context differences

Ladder stability

Distribution tail

• This is one 1,048,064-position shard, not the full 10,480,640-position run. The ten-shard run predates the histogram, so it could not be recomputed without re-running it.
• The quantiles are bin-bounded, not exact: each receipt carries lower / upper / estimate per quantile, with a relative bin width of about 5.6 %. The maxima and the exceedance counts are exact.
• The 10M receipts remain the source for the full-run means, intervals and paired results; nothing here replaces them.

What the v5 numbers do not say

• Absolute KLD is suite-specific. v5 values are not comparable to the v3 values in the next section: the corpus mix differs, and K4 reads 0.029679 there against 0.010604 here. Only within-suite ordering and paired differences transfer between the two.
• Cumulative percentiles come from one shard, not from all ten. The ten shard reports of the 10M run carry no token-level KLD histogram, so median/p95/p99/p999 could not be recombined across them. The tail table above closes that gap on shard 0 (receipts/kld5-1M-tail-*.json); across all 10,480,640 positions only the means, the intervals, the paired results and the exact global maximum exist.
• Captures: the reference survived, the candidates did not. The five candidates' hidden states and the BF16 references for shards 1-9 were deleted shard by shard to fit 135 GB of scratch, so unlike the v3 dataset this run is recomputable from the pinned corpus fetch log and suite manifest. The shard-0 BF16 reference was kept and is published, with the suite, all ten shard views and 79 per-shard reports, so a new candidate can be scored against the identical contexts without recapturing the reference — see Reproduce this.

Against GGUF, measured on our suite

1. 6 bits, Q6_K against hydrated: our sentence understated their byte spend roughly threefold. "+1.186 GiB more file" is +1.198 on tensors and +3.634 GiB of transformer body (23.1 % more than ours). Their fidelity win at 6 bits stands exactly as published - it is the price we mis-stated, in our own favour.
2. 6 bits, deployed: a multimodal Q6_K deployment is 22.18 GiB against our 20.13 GiB whole tree, so ours is 2.053 GiB smaller and needs no second file.
3. 5 bits, UD-Q5_K_XL against the context edition: this claim was wrong against us. We do not pay "0.445 GiB more" for the win - on transformer body they carry 2.148 GiB more (+14.4 %), and our deployed multimodal artifact is 0.422 GiB smaller.
4. 8 bits, Q8_0 against online K5/K6: the bodies agree to 1.7 % (24.526 against 24.119), the most format-comparable pair on the table, and they win it cleanly. That row is the reason the others are worth reading: this is not a table where every axis favours the author.

1. At the nominal 6-bit point, llama.cpp Q6_K measures 0.002035 and vLLM hydrated 0.002700.
2. At the nominal 5-bit point, the vLLM context edition measures 0.003409 and llama.cpp UD-Q5_K_XL 0.004444.

Cross-citation: the same three GGUFs under llama.cpp's own protocol

Prior receipt: v3 development suite (136 contexts, 278,392 positions)

Attention width is a runtime knob, and native context needs the K4 build

Measured on a real RTX 5090, by an independent tester

Prior receipt: v4 post-selection qualification (36 contexts)

Public capability — MMLU-Pro, item-paired against BF16

The pre-registered bar, and this build's verdict

Why a 70-item suite cannot certify this bar

Two protocol facts that bound the reading

• Exact-answer agreement is 0/70 for every EXL3 candidate, and 1/70 for official FP8 (one math item, a 113-token answer both models pass). Long chains of thought differ token-wise on essentially every item, so pass/fail outcome is the only meaningful pairing unit; nothing here is a generated-text match claim.
• Four BF16 items end at the 5,120-token completion cap with no letter emitted and are scored as failures under the plan's frozen addendum, so the control itself is depressed by the cap; per-model counts are in the table (this build: 4). The earlier 2,048-cap control, where BF16 lost 7/70 to truncation, is retained unchanged at receipts/public-capability-bf16-superseded-cap2048.json.

Status of this evidence

Downstream task retention — 40-task smoke suite (prior, narrower evidence)

Throughput

Concurrent serving: speculative depth is a concurrency-dependent choice

Serving

Recipe A — unmodified image, eager only

Recipe B — headline configuration (graphs + prefill dispatch)

1. --quantization exl3 is mandatory — auto-detection only fires for GLM-5.2 metadata.
2. Both performance features need unmerged patches, which is why recipe A and recipe B are separate. VLLM_EXL3_GRAPH_DECODE=1 + cudagraph_mode: FULL_DECODE_ONLY need PR #314 (without it the loader refuses non-eager execution and decode loses ~46-50 %). Reconstructed generic EXL3 prefill needs PR #316. The maintained module additionally uses PR #318 to route native K6/MCG B12X shards into that path at prefill row counts. Current upstream heads have no CI result: pre-run jobs are blocked by repository policy labels, not by a demonstrated failure.
3. The ignore list is mandatory and its anchoring is subtle. Prefixes carry no leading model., so re:.*visual\..* matches while re:.*\.visual\..* silently does not — and the wrong pattern crashes startup (#311, fixed by PR #312). The tested config also ignores mtp.* from the generic BF16 online overlay. EXL3 ownership is checked first, so the serialized quantized draft still uses the EXL3 loader; the ignore prevents accidental re-encoding only when a draft projection is not EXL3-owned.
4. --mm-processor-kwargs '{"truncation":false}' is required for images whose expanded token sequence exceeds 2,048, or requests fail with HTTP 400 (#313). Verified here: a 2044×1622 image answers correctly with the flag.
5. First load encodes 208 attention projections to K6 — measured 957 s cold against 173 s with a warm cache at an 8,192 window; the hydrated sibling, with nothing to encode, starts in 178 s (receipts/startup-times.json). Point VLLM_EXL3_ONLINE_CACHE_DIR at persistent storage.
6. Prefix caching is on in the recipe below, and it is on here and not everywhere. At an 8,192-token window the KV pool is roughly thirty times the window, so a single request comes nowhere near the pool ceiling and the failures that stopped the native-context profile cannot occur. That is not an argument, it is the reason this recipe was measured separately: it starts healthy on the promoted image with --enable-prefix-caching --mamba-cache-mode align, answers a text and an image request exactly, and reports enable_prefix_caching: True in the engine banner (receipts/production-image.json). The context edition's native 262,144-token recipe does not enable it, and its card explains why (receipts/qualification-5090-apc.json).
   The release unit moved on 2026-08-16, and #51113 is why. Upstream vLLM #51113 (mamba align prefill-chunk splitting: a chunk that ends mid-block leaves its slot holding a short state, which a later chunk then publishes anyway — wrong tokens, HTTP 200, no crash) merged 2026-08-06, after the pinned public image was built, and is still absent from it and from fork head fa033bd4e. Cherry-picks were requested upstream on 2026-08-16 (issue #392, PR #393). Until they land it is carried as tools/vllm-mamba-align-scheduler.py (sha256 b431c106…), and the image this project serves is now the four-module localhost/vllm:gg-r34-patched-apc, manifest sha256:16a936b877b90f…, promoted from the three-module localhost/vllm:gg-r34-patched (sha256:6eca4c693f01b6…) (receipts/production-image.json). With prefix caching off the two images are not merely similar but behaviourally identical, because the added module's changed function is unreachable unless mamba_cache_mode is align — so the earlier hardware qualification carries over unchanged. One warning if you inspect the image yourself: its build-time label io.malaiwah.image.qualified still reads false and is superseded by the receipts named here — it was written before the image could possibly have been qualified, and correcting it would add a layer and change the very digest that was measured. That digest is local to the build host, so the recipes here reproduce its content with sha256-verified read-only mounts over the pullable public base.
   What prefix caching buys. On disjoint documents, so the cold case is genuinely cold: a 32,842-token prefix went 12.07 s cold → 1.04 s warm (11.6×, 2,442 of 32,842 prompt tokens recomputed, 92.6 % hit rate) and a 131,146-token prefix went 67.60 s → 2.31 s (29.3×, 3,146 of 131,146 recomputed, 97.6 %); a 38-request schedule ran 84.0 s with the cache against 144.4 s without (receipts/apc-poison-repro.json). And what we actually know about its safety. Correctness was probed adversarially before any of this shipped: seven freshly started servers, 38 requests each, 266 scored requests, nested token prefixes so later requests hit blocks published by earlier ones, and no prompt length a multiple of the measured 1,600-token mamba block, so prefill chunks end mid-block by construction — zero corrupted responses, zero wrong answers, zero acceptance collapses, on the unpatched image as well as the patched one. Thresholds were committed before the first server started; the worst repeated block was 15 characters against an 80-character threshold, with no U+FFFD anywhere. Greedy chosen-logprob drift with the cache on is 0.1063 mean absolute against a measured run-to-run floor of 0.0823 — drift, never an answer change. So the module is carried as insurance backed by upstream's own regression file — 14 failed / 6 passed against the vendored scheduler, 20 passed against this one — and not by a reproduction of our own: we tried hard to reproduce the reported corruption and could not (receipts/mamba-align-defect.json).
   LMCache is unmeasured by us. It is not part of any recipe on this card, this project has never run it, and it is the outstanding suspect in the one user report of prefix-cache corruption we have. Nothing here says LMCache is safe; the evidence above covers vLLM's own prefix cache and nothing else.
   #51812 is now recommended for this recipe, and it stays an overlay. Upstream #51812 (Qwen GDN speculative gate ordering: the vendored code gathers the speculative Q/K/V rows but hands the recurrent update the ungathered a/b gate tensors, so gate row i can belong to a different token than Q/K/V row i) merged 2026-08-11 and is absent from the promoted image. Whether that path is ever entered is no longer an argument: a CPU-only counter mounted over the engine's GDN metadata builder measured it, and in the flag set Recipe B ships — window 8,192, --gpu-memory-utilization 0.85, --enable-prefix-caching --mamba-cache-mode align, MTP-3, at eight concurrent streams (the arm additionally pinned fp8 KV and --max-num-batched-tokens 2048) — it was entered: three events in 5,825 metadata builds, 0.515 per thousand, over 468 requests with zero errors and a prefix-cache hit rate reaching 50.1 % (receipts/gdn-gate-concurrency.json). That arm served the context edition's weights: the counter reads only the scheduler's host-side arrays, so what it measures is a function of flags and traffic, with the checkpoint entering only through the size of the KV pool. The mechanism, in one sentence you can act on: at eight streams with prefix caching on, a short non-speculative request can land between speculative ones in the same batch, and the unpatched gather then misaligns the gates. All three firings had one shape — six speculative decodes plus one non-speculative request, that request beginning at token index 20 and displacing one four-token speculative decode, whose four gate rows were read from the wrong tokens. The same instrument saw zero events in 3,329 builds at eight streams with prefix caching off (below 0.90 per thousand, 95 % upper bound) and zero in 8,065 builds at a quarter of the token budget, so it is the cache path that opens this, not concurrency alone.
   So mount it if you serve Recipe B with more than one sequence in flight: tools/vllm-qwen-gdn-spec-gates.py (sha256 7cd3f5fe763b621048af4817951a841d99c8b700d9a56ded27ccaca5a56ccbe0) read-only over /opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py. It is diff-identical to upstream's eight changed lines and py_compile-clean under the image's Python 3.12.3. It is an overlay and deliberately not part of the qualified digest: sha256:16a936b877b90f… is what was qualified, and a reachability count is not evidence that would survive a re-qualification, so it is mounted over the vendored file rather than promoted into the image. At --max-num-seqs 1 no mixed batch can form at all and none of this applies.
   The effect on answers was not measured, and measuring it was declined on resolution grounds. Three events in 5,825 builds cannot move a statistic whose run-to-run floor is 0.0823 mean absolute chosen-logprob error against a per-event effect of 0.002755 — a noise floor about thirty times the size of one event — so an A/B would have returned its own noise and was deliberately not run. Nothing here claims the overlay changes an answer, or that it does not. The recommendation follows a rule fixed before the GPU window opened instead: a nonzero rate in a regime we ship gets the free fix, because a silently miscomputed forward pass gives the operator no signal at all. Traffic in that arm was adversarial by design — eight speculative streams plus injected short prompts and full-cache-hit repeats — so 0.515 per thousand is an upper bound on a shipped regime, not a forecast for your workload (receipts/gdn-spec-gate-defect.json is the source-level defect analysis).
   Scope on this build. Recipe B above is smoke-proven with prefix caching on — it starts and answers a text and an image request exactly on the promoted image (receipts/production-image.json) — but the nine-gate qualification was run on the native-context profile, not on this 8,192-token one. Recipe A stays prefix-caching-off and must: it is the unmodified public image, which predates #51113.

KV-cache dtype: fp8 is the family's measured default

Reconstruct-scratch arena: +17,874 KV tokens, and it stays an overlay

Client contract, inherited unchanged from upstream

Optional: narrow the input embedding table

Chat template

• reasoning_effort accepts only xhigh (default), medium and low - measured against a live endpoint, correcting an earlier revision of this card that also listed none. The server answers HTTP 400 Unexpected reasoning effort none. Supported types are xhigh (default), medium, and low. for none, exactly as it does for high, minimal and max - all of which vLLM's OpenAI surface advertises and this template rejects. If your client hard-codes high, serve with --default-chat-template-kwargs.reasoning_effort=xhigh.
• What the ladder costs, measured rather than described. Same prompt, temperature 0, 32,768-token budget so no arm is truncated, two repeats each, on the qualified 1x endpoint (receipts/reasoning-effort-1x.json):

  reasoning_effort	completion tokens	reasoning chars	latency	finish	correct algorithm
  (unset)	18,085	56,955	168.9 s	stop	2/2
  xhigh (default)	18,085	56,955	168.8 s	stop	2/2
  medium	3,436	4,767	26.2 s	stop	2/2
  low	3,517	4,606	27.0 s	stop	2/2

  Three things follow. Unset is byte-identical to xhigh, which is how the default is now measured rather than assumed. xhigh costs 5.3x the output tokens and 6.4x the wall clock of medium for the same correct answer on this task - the single largest cost knob on the client side of this card, and the one to reach for first in any agent loop bounded by a per-task timeout rather than by quality. And low is not cheaper than medium (3,517 against 3,436 tokens): the bottom of the ladder is flat, so medium is the useful floor. Read narrowly: one prompt, one shape, an exact-match check on whether the right algorithm is named - it establishes token cost, not a quality ordering.
• Exactly one system message, first. Two system messages 400 with System message must be at the beginning. (vllm#41114, open; fix PR vllm#44505 closed unmerged). Merge them client-side. A developer role is safe - vLLM folds and consolidates it (vllm#43590).
• Content blocks must be text, image, image_url or video. Anything else, including an Anthropic tool_result carrying a tool_reference item on /v1/messages, 400s with Unexpected item type in content. (vllm#52489, open).
• Only the leading system message is accepted. A second or mid-conversation system role raises. Clients must merge system instructions into the first system message before rendering history.
• Tool-call argument values are an unescaped XML boundary. Before replaying assistant history, reject values containing literal </parameter> or <parameter= delimiters. Treat replayed calls and tool output as untrusted data; the template cannot escape those sequences.

Verification performed, and not performed

Safety and intended use

Reproduce this

Prior art and credits

• turboderp / turboderp-org/exllamav3 — the EXL3 Trellis format, LDLQ calibration, MCG codebook, and the conversion pipeline.
• Martin Vit / voipmonitor / festr2 (local-inference-lab) — the Gilded Gnosis vLLM fork, B12X Trellis kernel integration, the voipmonitor/vllm Docker images, the rtx6kpro serving wiki, and the exl3-b6 online-encoding overlay whose distribution-fidelity protocol this evaluation follows.
• Luke Alonso / lukealonso — B12X kernel co-author (PR #31, NF3/NVFP4) and the NVFP4 quantization methodology whose protection patterns informed the recipe design.
• Brandon Music / brandonmusic — the ShapleyMCG expert-bit allocation method, the EXL3 checkpoint patterns (TR3 schema, online K6, shared-expert handling) that this recipe family is modelled on, and the exllamav3 SM120 retile fork.
• Josh Cartu / jcartu — the MTP78 recipe and rank-sliced MTP runtime that informed malaiwah/GLM-5.2-EXL3-TR3-MTP78: the graft/encode/serve procedure for the MTP draft layer, plus the vLLM normalize_rank_sliced_weight_name() loader patch.
• Qwen — the base model (Qwen/Qwen3.8-27B, Apache-2.0) and its official FP8 derivative (Qwen/Qwen3.8-27B-FP8).
• nvidia/Qwen3.6-27B-NVFP4 — the role-aware recipe this design is modelled on (4-bit MLP, 8-bit attention, protected embeddings/vision/MTP).
• unsloth/Qwen3.8-27B-NVFP4 — independent confirmation of the same protection pattern.
• malaiwah / progressive-tensors — research, recipe, harness and receipts. Per-expert EXL3 provenance work, the per-bit error ladder, the GLM-5.2-EXL3-TR3-MTP78 checkpoint, and the RTX 5090 context-capacity qualification that caught a 262K overclaim.

Companion repository