Qwen3.8-27B INT4 AutoRound
A
W4A16 (INT4 weight, FP16 activation) quantization of
https://huggingface.co/Qwen/Qwen3.8-27B, produced with
Intel's AutoRound.
Built as a
drop-in model slot for the club-3090 dual-RTX-3090 vLLM stack — format-identical to
https://huggingface.co/Lorbus/Qwen3.6-27B-int4-AutoRound, whose recipe this replicates for the new 3.8 release.
TL;DR
- Base: Qwen3.8-27B (27B dense VLM, hybrid Gated DeltaNet + full attention, Aug 2026)
- Quant: INT4 W4A16, group_size 128, symmetric
- Tool:
auto-round 0.14.2 (default recipe, 200 iters, torch.compile)
- Size: ~19.6 GB (down from ~54 GB BF16) — ~2.7x reduction
- MTP preserved: the entire Multi-Token Prediction head (
mtp.fc + all mtp.layers.0 linears) is kept in BF16, enabling native speculative decoding in vLLM — 1.55x measured decode throughput (127 vs 82 tok/s, single stream, RTX 5090) at 0.69 greedy draft acceptance on vLLM 0.27.1
- Purpose: the stock club-3090 dual compose serves this repo with a two-line change
Quantization details
| Field | Value |
|---|
| Base | Qwen/Qwen3.8-27B (BF16 — never quantize from the FP8 repo) |
| Method | AutoRound (intel/auto-round), default recipe |
| Scheme | W4A16 (4-bit weights, FP16 activations) |
| Bits | 4 |
| Group size | 128 |
| Symmetric | yes |
| Packing format | auto_round:auto_gptq |
| Unquantized layers | linear_attn.in_proj_a/b, the full MTP head (mtp.fc, mtp.layers.0.* linears), all LayerNorms/RMSNorms, router gates, vision tower, embeddings, lm_head |
| Calibration samples | 128 (default) |
| Iterations | 200 |
| torch.compile | enabled |
| GPU used for quant | 1× RTX 5090 (32 GB, SM120), low_gpu_mem_usage=True |
| auto-round version | 0.14.2 |
| Quant wall time | ~1h 10m (peak VRAM 27.7 GB, peak RAM 23.7 GB) |
Unquantized layers — why
linear_attn.in_proj_a/b: low-rank projections in the Gated DeltaNet blocks; shapes not divisible by 32 (group_size), so AutoRound skips them. Tiny parameter fraction.mtp.fc: the Multi-Token Prediction fusion layer — see the fix below.- Norms, routers, vision tower, embeddings: precision-sensitive and small; kept at base precision (BF16), matching the reference quant and the official FP8 repo's own exclusion list.
MTP fix — what's different from a vanilla AutoRound run
A plain auto-round run on this architecture packs mtp.fc as INT4. In that form, vLLM's Qwen3_5MTP loader skips the layer entirely (it expects fc.weight, finds fc.qweight), and MTP speculative decoding silently runs at 0% acceptance.
This build goes one step further than the 3.6 reference quant and keeps the entire MTP head in BF16:
mtp.fc: excluded from quantization natively via --fp_layers mtp.fc (auto-round ≥ 0.14 supports this), so it was never packed. Recorded in quantization_config.extra_config as 16-bit fp.mtp.layers.0 linears (7): auto-round 0.14.2's exporter RTN-packs these (untuned) at export time; they were replaced with the original BF16 weights from the base model post-hoc (single-shard surgery in model_extra_tensors.safetensors) and marked as 16-bit fp in extra_config.
The whole head is only ~220 MB, so size impact is negligible. Result: MTP works out of the box. If you serve without --speculative-config, the BF16 MTP weights are simply unused and harmless.
Validation (RTX 5090, vLLM 0.27.1)
| Gate | Result |
|---|
| Code generation | PASS — correct, coherent Python |
| MTP acceptance | PASS — 0.69 greedy / ~0.58 sampled (non-thinking) / ~0.47 (thinking, temp 1.0); fail mode ≈0 excluded. An A/B test (INT4 vs BF16 MTP internals) scored identically, so the gap vs older ~0.8–0.9 reports is a vLLM-version difference, not quant quality |
| Decode speedup | 1.55x (900-token completion: 7.1 s with MTP vs 11.0 s without, single stream) |
| Vision | PASS — correct scene description of an unlabeled photo |
Note: on a single desktop-shared 32 GB GPU, a 131072 ctx OOMs at KV-cache init (hybrid-GDN state pool scales with max_num_seqs); validated at 32768 ctx / 32 seqs. The dual-3090 TP=2 target rig has sufficient headroom for full context.
Known issue — vLLM 0.27.x GDN+MTP engine bug (not a checkpoint defect)
On vLLM 0.27.x, the hybrid-GDN + MTP speculative-decoding path is unstable when
prefix caching and/or
async scheduling are enabled: the engine either crashes (
CUDA illegal memory access) or — reported at TP=2 —
permanently collapses MTP draft acceptance to 0% after a long session once any sequence crosses position 32,768, surviving context resets until restart (
vllm#52873, closed as not-planned; crash family #37035/#34948).
This was investigated against this checkpoint: the artifact is bit-identical to the upload (sha256-verified), and with --no-async-scheduling --no-enable-prefix-caching the BF16 MTP head drafted at 0.79–1.00 windowed acceptance through 37k+ generated tokens and past position 32,768 with no collapse. The failure modes are engine-side, not weight-side; checkpoints with quantized MTP heads hit the same engine bugs.
Serving guidance: with the MTP drafter enabled, add --no-async-scheduling --no-enable-prefix-caching (cost: shared-prefix re-prefill). Without the drafter (num_speculative_tokens=0), prefix caching and async scheduling are safe. Capping --max-model-len below 32,768 does not make the unstable path safe — crashes were observed at ~21k ctx.
Quantization environment
python 3.12 · auto-round 0.14.2 · transformers 5.15.0 · torch 2.13.0+cu130 · quantized from a local BF16 checkout of Qwen/Qwen3.8-27B
Update 2026-09-06 — drop-in status for the club-3090 stack, and the collapse reproduced (engine-side)
TL;DR for anyone serving this checkpoint on dual RTX 3090s: the weights are fine — the vLLM 0.27.x GDN+MTP bug above is what kills long sessions, and on 2026-09-06 it was finally reproduced on this exact artifact, as a hard crash. Until vllm#50021 lands in a release you can pin, treat this checkpoint as a drop-in model slot only at MTP n=3 (or with the drafter off). Do not run MTP n=4 on stock vLLM 0.27.1.
What the re-test showed
Serving config under test: W4A8 int8 activations (see recipe below), prefix caching ON, async scheduling ON, fp8 KV, FlashInfer, TP=2 — i.e. the club-3090 dual-fast feature set. Identical growing multi-turn conversation each run (the same driver as the August investigation, greedy, thinking off):
| Run | MTP n | Outcome |
|---|
| A | 3 | Clean — 39,900 cumulative generated tokens (2.7× the reported ~14.5k trigger), ctx to 44k, windowed acceptance 0.87–0.98 throughout, no errors |
| B | 4 | Engine crash at ~14,950 cumulative generated tokens — CUDA error: an illegal memory access was encountered at gpu_model_runner.py:3875/3877 synchronize_input_prep, Xid 31 MMU fault on both GPUs simultaneously, acceptance still healthy (0.91–0.98) at death |
Run B matches, point for point, the crash that club-3090 filed as
#1052 against this checkpoint's deployment: same threshold (~14.5k cumulative gen with async scheduling), same signature (dual Xid 31), same fault site (
synchronize_input_prep — the exact frame their
vllm-gdn-mtp-async-spec-order patch gates, upstream
vllm#50021, still open). The same artifact at n=3, and at n=4 on W4A16 in August, ran past every threshold with the drafter healthy — so the discriminator is the
serving configuration, not the weights.
Why the "checkpoint-specific" attribution doesn't hold: the buggy code derives an array index from the per-step accepted-token count without bounds-checking it. A checkpoint only changes the rhythm of those counts (a BF16 drafter head and a quantized one draft with different acceptance patterns), which moves where the race tends to fire — it cannot cause or prevent it. club-3090's own tracker agrees on the substance: the crash "reproduc[es] across two unrelated quant formats" (their #838), their replacement checkpoint later showed the same decay signature (#1096), and the fix they ship for n=4 is an engine patch, not a checkpoint. Frozenlock differs from this quant mainly in quantizing the MTP head and in calibration data — neither is a plausible mechanism for a bounds-check bug in vLLM's spec-decode path, and neither survives the A/Bs above as an explanation.
The recipe that works (validated 2026-09-06, 2×3090 TP=2, vLLM 0.27.1)
1export VLLM_MARLIN_INPUT_DTYPE=int8 # W4A8: int8 activations on the Marlin int4 GEMMs
2export VLLM_CACHE_ROOT=~/.cache/vllm-w4a8 # torch.compile cache cannot key on W4A8 — isolate it
3vllm serve /path/to/this-checkpoint \
4 --quantization auto_round --dtype float16 \
5 --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.95 \
6 --max-num-batched-tokens 16384 --kv-cache-dtype fp8 \
7 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
8 --enable-prefix-caching --attention-backend flashinfer \
9 --override-generation-config '{"temperature":0.7,"top_p":0.80,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}' \
10 --default-chat-template-kwargs '{"enable_thinking": false}' \
11 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
- W4A8 needs two vLLM source patches on 0.27.1 — stock silently ignores the toggle for auto-round checkpoints, and the int8 kernel mis-reads AutoRound's negative group scales (garbage output). Both are small, idempotent, anchor-checked Python patchers, available as club-3090's
w4a8-int8-act (patches: thread input_dtype through the inc route; fold negative scales into the weight codes at load). Measured single-variable A/B on this checkpoint: prefill 1359 → 1860 tok/s @10K (+36.9%), TTFT −27%, decode neutral, KV pool −14% (503,785 → 434,954 tokens). Rollback = unset the env var; the patches are runtime no-ops without it.
- Crash avoidance: n=3 is the tested depth. If you need n=4, either apply club-3090's
vllm-gdn-mtp-async-spec-order + vllm-flashinfer-decode-pin patches (their fix for exactly the fault site above), or run with --no-async-scheduling (their #1059 A/B: eliminates the crash at ~0% single-stream cost). SPEC_N=0 is the always-safe fallback.
- Transient
expandable_segments allocator OOM-retries during weight load at util 0.95 are normal under W4A8 (the fold repacks per layer); if a boot ever dies there, lower --gpu-memory-utilization a notch.
Drop-in note for club-3090 compose users: this checkpoint works as the model slot with the usual two-line change, but set SPEC_N=3 on their dual-fast tier (its n=4 default assumes their vendored spec-decode patches). Artifacts for the 2026-09-06 re-test (per-turn logs, CSVs, crash journal with kernel Xid lines) are available on request.
