Qwen3.8-27B-heretic-ara — mixed INT4 + INT8 GPTQ
TL;DR — 22 GB mixed-precision GPTQ checkpoint (INT4 MLP + INT8 attention projections, BF16 for the rest). Measured −0.76pp on GSM8K and +2.5% perplexity vs the BF16 original — i.e. effectively lossless — while cutting weight memory ~2.4× vs BF16. Loads directly in vLLM ≥ 0.27 with quantization=auto_gptq.
A data-driven mixed-precision GPTQ quantization of the heretic-ara fine-tune:
| Component | Precision |
|---|
| MLP gate/up/down × 64 | INT4 GPTQ (group 128, sym) |
GDN in_proj_qkv/z/out_proj × 48 (recurrent linear-attention) | INT8 GPTQ (group 128, sym) |
Full-attn q/k/v/o_proj × 16 | INT8 GPTQ (group 128, sym) |
GDN in_proj_a/b, conv1d, A_log, dt_bias, all norms | native BF16 |
| Vision tower, lm_head, embed_tokens | BF16 |
| MTP head (15 tensors) | BF16 |
Quantizer: GPTQModel 7.3.4 (data-driven, 607 calibration prompts from the
fine-tune's own corpora). The mixed-precision layout is driven by the dynamic
field of quantization_config, which both GPTQModel and vLLM ≥ 0.27 parse
natively (-:regex = keep unquantized, +:regex = per-module bit override),
so the checkpoint is vLLM-loadable as-is.
Quality vs BF16 (teacher-forced full-vocab KLD)
| Suite | this build | refs: W8A16 / FP8 |
|---|
| short 8-prompt (464 pos), nats/token | 0.0200 | 0.0009 / 0.0019 |
| long 4096-token probe, nats/token | 0.0895 | — / 0.0044 |
| top-1 long | 98.4% | 99.4% / 98.5% |
Downstream benchmarks (vLLM 0.27.1, A100 80GB)
Run against the BF16 original with the model's default sampling
(temperature=1.0, top_k=20, top_p=0.95), identical prompts/subsets for both.
GSM8K (5-shot, full 1,319-question test set)
| Metric | BF16 original | this build | Δ |
|---|
| exact match (flexible extract) | 78.39% | 77.63% | −0.76pp |
| exact match (strict) | 76.80% | 76.65% | −0.15pp |
| Empty responses (immediate EOS) | 1.6% | 7.4% | +5.8pp |
The small gap is driven almost entirely by a higher early-EOS rate (the
quantized model occasionally stops right after the Answer: prompt at
temperature 1.0); conditional on producing a non-empty answer the two models
are statistically indistinguishable. If this matters for your workload, lower
the temperature (≤ 0.7) or retry empty completions.
Perplexity — C4 validation (30k-doc subset, 293,618 tokens)
| Model | PPL (lower = better) |
|---|
| BF16 original | 11.145 |
| this build | 11.422 |
| Δ | +2.48% |
Comfortably inside the <5% threshold typically considered safe for aggressive
(4-bit) quantization.
Verified on vLLM 0.27.1
quantization=auto_gptq; mixed 4/8-bit modules resolved per-module
(Marlin 4-bit for MLP, 8-bit kernel for the projections)- text generation, vision (image Q&A), MTP speculative decoding all working
(acceptance stats in the run log)
Why the config has fused-name rules
vLLM fuses some separate checkpoint weights on load (qwen3-next-style):
in_proj_qkv+in_proj_z → in_proj_qkvz, in_proj_b+in_proj_a →
in_proj_ba, q/k/v_proj → qkv_proj, gate/up_proj → gate_up_proj.
vLLM resolves each module's precision from the dynamic field using the
fused name, so this config adds rules for the fused names
(+:.*in_proj_qkvz$ → 8-bit, -:.*in_proj_ba$ → keep BF16,
+:.*qkv_proj$ → 8-bit) on top of the per-module GPTQModel rules.
Without them vLLM creates the fused GDN module at the base 4-bit width and
fails to load the 8-bit shards (shape assert). The weights themselves are
exactly the GPTQModel output; only the JSON config was extended.
Serve
1vllm serve <this-dir> \
2 --dtype bfloat16 --max-model-len 262144 --gpu-memory-utilization 0.92 \
3 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
4 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
5 --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}'
