A latency-optimized INT4 (W4A16) build of Qwen/Qwen3.8-27B for interactive, thinking-off serving on vLLM: MTP speculative head kept intact and measured, lm_head in INT8, plus the serving flags that make this hybrid GDN+attention model's prefix cache actually hit.

Why this build. Most 4-bit checkpoints of this model are made and judged for throughput or benchmark accuracy. This one targets the other regime — real-time conversation (voice agents, embodied assistants, chat UIs) where a single GPU must turn a ~1.6 k-token prompt into a 20–60-token reply with very low time-to-first-token and per-turn latency, thinking off, one or a few streams at a time. The calibration data, the preserved MTP draft head, the INT8 output layer and the serving flags below were chosen for that use, and the figures on this page are measured end-to-end turn latencies rather than peak throughput.

TL;DR — 18.2 GB (BF16 base 55.6 GB) · INT4 symmetric g128, compressed-tensors → vLLM Marlin kernel on any SM 8.0+ GPU · vision tower + MTP draft head BF16, lm_head INT8 · zero-config in vLLM ≥ 0.17 (tested 0.24.0 / 0.27.1) · MTP verified working (64–73 % draft acceptance on chat traffic) · end-to-end turn latency under 0.5 s for a ~20-token reply after a 1.6 k-token prompt on one RTX 6000 Ada (415 ms) or one RTX 5090 (~280 ms), TTFT under 100 ms with the recipe below — single stream, same harness for every number on this page.

Naming note — "AWQ" is the checkpoint class people search for (4-bit W4A16, same format and kernel path as AWQ-INT4 checkpoints). We built the same model with three algorithms on the same calibration data — AutoRound, AWQ and GPTQ — and measured them side by side; main ships AutoRound + INT8 lm_head, the AWQ and GPTQ builds are the awq-llmcompressor / gptq-llmcompressor branches (see Which algorithm). Serving speed is identical across the three by construction.

Quick start (vLLM ≥ 0.17)

Recipe A — shortest turn latency for chat with a long fixed system prompt

bash

vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency \
  --served-model-name qwen38 --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

Size the fixed part of your prompt to a multiple of the cache block (see Prefix caching) → TTFT 67 ms, 415 ms per 18-token turn on one RTX 6000 Ada.

Recipe B — longer replies (≳ 25 tokens) or short prompts: add MTP

bash

vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency \
  --served-model-name qwen38 --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(model="qwen38", max_tokens=128, temperature=0.7,
    messages=[{"role": "system", "content": "You are a concise, friendly assistant."},
              {"role": "user", "content": "I've had a long day. Any quick tip to unwind?"}])
print(r.choices[0].message.content)

No --quantization flag needed; vLLM logs Using MarlinLinearKernel for CompressedTensorsWNA16.
Check MTP is really accepting: curl :8000/metrics | grep spec_decode → accepted/draft ≈ 0.6–0.75 on chat. A drafter that accepts 0 % is slower than MTP off.
Don't pass --calculate-kv-scales on hybrid GDN models (vllm#37554). SGLang: untested.
Tokenizer and config files are byte-identical to upstream Qwen/Qwen3.8-27B.

Latency (measured)

Conversational prompts (≈1.6 k-token system prompt + 2 turns of history + a user utterance), greedy, thinking off, 18 prompts × 3 runs, streamed over HTTP, medians; "ms/tok" = inter-token latency. The headline numbers are for ~20-token replies; from the measured TTFT and per-token rates, a 40–60-token reply lands at roughly 0.8–1.1 s on the RTX 6000 Ada and 0.45–0.6 s on the RTX 5090 with MTP on.

Serving-recipe ladder — RTX 6000 Ada 48 GB, vLLM 0.24.0, MTP off

Configuration	Prompt tok	Cache hit	TTFT	Turn	ms/tok
default (`--enable-prefix-caching`)	1449	51 %	272 ms	634 ms	20.4
+ `--mamba-ssm-cache-dtype bfloat16`	1449	79 %	120 ms	468 ms	20.1
+ block-aligned system prompt (1604 tok)	1662	90 %	75 ms	418 ms	20.1
+ both	1662	92 %	67 ms	415 ms	19.9
ref: Qwen3.6-35B-A3B 4-bit (MoE, 3B active), same card/flags	1449	67 %	95 ms	413 ms	15.0

Same weights and identical outputs in every row — the 634 → 415 ms is purely cache behaviour.

MTP — RTX 6000 Ada, vLLM 0.24.0: K=0 20.4 ms/tok → K=1 15.9 (79 % acceptance) → K=2 13.7 (73 %, 1.49×) → K=3 13.5 (61 %). With K=2 + bf16 state + block-aligned prompt: TTFT 205 ms, 455 ms/turn, 72 % cache hit — MTP overtakes Recipe A once replies exceed ~25 tokens. Calibration data matters here: the same recipe calibrated on pile-10k reaches 57.9 % acceptance, on conversational transcripts 73.1 %; a community export without mtp in ignore loads a broken drafter and runs at 0 % acceptance, 1.5× slower than MTP off.

Head precision, fidelity and concurrency — RTX 5090 32 GB, vLLM 0.27.1 (NLL = per-token negative log-likelihood of the BF16 base's own greedy answers under the quantized model; lower = closer to BF16)

Variant	MTP off TTFT / turn / ms·tok	MTP K=2 TTFT / turn / ms·tok	K=2 accept	NLL	Repetition / vision / format probes
INT4 body, BF16 lm_head	80 / 300 / 11.6	145 / 299 / 8.7	64.6 %	0.327	all pass
INT4 + INT8 lm_head — `main`	78 / 281 / 10.8	140 / 274 / 7.6	64.4 %	0.326	all pass
INT4 + INT4 lm_head — `lm-head-int4`	76 / 254 / 10.4	136 / 262 / 7.0	66.0 %	0.326	all pass
MTP head also INT4 — `mtp-int4-autoround`	81 / 280 / 11.6	144 / 286 / 8.0	67.8 %	0.324	all pass

The levers that move decode latency cut bytes per token: INT8 lm_head −13 %, INT4 lm_head −20 %, INT4 MTP head −8 % (MTP on). Concurrency (5 streams, MTP off): 394 ms/turn, 253 tok/s aggregate; with MTP K=2: 750 ms/turn — MTP is a single-stream optimisation, turn it off for multi-user endpoints.

Prefix caching on a hybrid GDN model

Qwen3.8-27B has 16 attention + 48 Gated-DeltaNet layers. vLLM forces the attention block size to ceil(GDN state bytes / KV bytes per token) — 784 tokens with the default fp32 state — and the prefix cache only reuses whole blocks:

--mamba-ssm-cache-dtype bfloat16 halves the state → 400-token blocks (51 % → 79 % hit, no output change on this workload);
size the fixed prompt just past a block multiple so only the per-turn tail is recomputed (→ 67–75 ms TTFT);
MTP + prefix cache: Qwen3.5-family only supports --mamba-cache-mode align, which stores GDN state only when a step ends on a block boundary — with speculation the hit rate can fall to 0 % unless you use both tricks above (then 72 %).

Which algorithm

Same base model, same calibration pool, same harness, same box (RTX 5090, vLLM 0.27.1); all rows with INT8 lm_head:

Algorithm (tool)	Calibration	MTP off TTFT / turn / ms·tok	K=2 accept	NLL mean / median	Verbatim / first-sentence match with BF16 (of 18)
AutoRound 0.14.2, 200 iters — `main`	384 × 2048	112 / 293 / 10.6	66.6 %	0.325 / 0.320	8 / 11
AWQ (llm-compressor 0.13.0) — `awq-llmcompressor`	256 × 2048	114 / 307 / 10.6	64.7 %	0.347 / 0.322	4 / 7
GPTQ (llm-compressor 0.13.0, `actorder=weight`) — `gptq-llmcompressor`	384 × 2048	112 / 294 / 10.6	67.0 %	0.328 / 0.342	5 / 8

Serving speed is identical (same tensors, same kernel). Fidelity to BF16 is best for AutoRound, GPTQ close on NLL, AWQ slightly behind (a few prompts raise its mean; its median is in line; llm-compressor's AWQ mappings leave o_proj / linear_attn.out_proj unscaled). Treat the ordering as indicative — 18 prompts, coarse verbatim counts, flat MTP acceptance — not as a benchmark. (This box reads ~30 ms higher TTFT than the one used in the table above for the same main weights; compare within one table.)

What is quantized, what is not

Module	Precision	Why
`self_attn.{q,k,v,o}_proj` (16 layers), `mlp.{gate,up,down}_proj` (64), `linear_attn.in_proj_qkv / in_proj_z / out_proj` (48)	INT4 sym g128	bulk of the bytes
`linear_attn.in_proj_a`, `in_proj_b` (48-wide gates)	BF16	4-bit hurts the recurrence, saves nothing, breaks Marlin under TP
Vision tower `model.visual.*`	BF16	keeps image/video quality
*MTP head `mtp.`** (15 tensors)	BF16, `re:.mtp.` in `ignore`	the draft must match the target closely or MTP never accepts
`lm_head`	INT8 sym g128 (`main`), INT4 (`lm-head-int4`)	1.27 B params read every token; INT8 is lossless on our metrics
embeddings, norms, `conv1d`	BF16	not on the decode-bandwidth path

Sizes: main 18.2 GB · lm-head-int4 17.6 GB · mtp-int4-autoround 19.0 GB · branches 18.2 GB. A 24 GB card serves text with a modest KV budget; 32 GB+ is comfortable at 12 k context.

Calibration & reproduce

530 sequences ≥ 2048 tokens rendered through the model's chat template with thinking off: ~60 % multi-turn conversational transcripts (assistant-persona system prompt + short colloquial turns, predominantly Chinese), ~20 % the same turns without a system prompt, ~20 % ShareGPT-GPT4 / UltraChat (zh/en). No benchmark test sets; the transcripts are not distributed and nothing from them is embedded in the weights. If your traffic differs (code, long reasoning), re-run the recipe on your own data — that is what moved MTP acceptance from 58 % to 73 % here.

bash

# AutoRound (main)
auto-round --model Qwen/Qwen3.8-27B --scheme W4A16 --bits 4 --group_size 128 \
  --iters 200 --nsamples 384 --seqlen 2048 --batch_size 4 \
  --dataset "/path/calib.jsonl:apply_chat_template=false" \
  --ignore_layers "model.visual,linear_attn.in_proj_a,linear_attn.in_proj_b,re:.*mtp.*" \
  --format llm_compressor --enable_torch_compile --output_dir out/      # 1 h 14 min on one RTX 6000 Ada

python

# AWQ / GPTQ (llm-compressor 0.13.0)
ignore = ["re:.*visual.*", "lm_head", r"re:.*linear_attn\.in_proj_a$", r"re:.*linear_attn\.in_proj_b$"]
recipe = [AWQModifier(duo_scaling=True), QuantizationModifier(ignore=ignore, scheme="W4A16", targets=["Linear"])]   # AWQ
# recipe = [GPTQModifier(ignore=ignore, scheme="W4A16", targets=["Linear"], dampening_frac=0.01, actorder="weight")]  # GPTQ
oneshot(model=model, processor=tokenizer, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained(out, save_compressed=True, max_shard_size="4GB")

Three things the exporters do not do for you: (1) keep the 15 BF16 mtp.* tensors in the checkpoint and add re:.*mtp.* to quantization_config.ignore — without both, vLLM's drafter loads garbage and MTP runs at 0 %; (2) copy the upstream tokenizer files back verbatim — save_pretrained in transformers 5.14/5.15 re-serialises tokenizer.json with an older pre-tokenizer regex (drops \p{M}, changes tokenization of scripts with combining marks); (3) the INT8 lm_head is applied post-hoc (RTN g128 via compressed_tensors, a group_1 scheme targeting re:.*lm_head$) because AutoRound's --quant_lm_head needs the 248 k-vocab logits in memory.

Revisions

Revision	Content	Load
`main`	INT4 body (AutoRound) + INT8 lm_head, MTP head BF16	compressed-tensors, zero-config (vLLM 0.24.0 / 0.27.1 verified)
`lm-head-int4`	INT4 body (AutoRound) + INT4 lm_head (fastest decode)	compressed-tensors
`mtp-int4-autoround`	INT4 body (AutoRound) + INT4 MTP head (`mtp.fc` BF16), lm_head BF16	`auto_round:auto_gptq` packing, vLLM `inc` backend, no flag
`awq-llmcompressor`	INT4 body (AWQ) + INT8 lm_head	compressed-tensors
`gptq-llmcompressor`	INT4 body (GPTQ) + INT8 lm_head	compressed-tensors

vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency --revision lm-head-int4 .... All revisions carry upstream-identical tokenizer/config files (earlier uploads had a transformers-resaved tokenizer.json).

Limitations

Built and measured for short-reply, thinking-off, single/few-stream interactive serving; long chain-of-thought and > 32 k-context behaviour were not evaluated beyond the base model's claims. No academic benchmarks were run; the fidelity numbers above are relative to the BF16 base on conversational prompts.
MTP roughly doubles per-turn latency at 5 concurrent streams; FP8 KV cache is not worth it here (only 16 of 64 layers keep KV).
vLLM 0.27.1 on Python 3.10 needs from __future__ import annotations at the top of flashinfer/comm/fd_exchange.py; fresh machines need g++ and python3-dev for the JIT kernels.

Acknowledgements

Qwen team for Qwen3.8; Intel for AutoRound; the vLLM / llm-compressor / compressed-tensors maintainers.