An independently re-quantized AWQ INT4 build of Qwen/Qwen3.5-4B with all linear
projections quantized — self-attention (8 full-attention layers), the gated
delta-net / linear-attention projections (24 layers), and the MLPs — while keeping
in_proj_a/in_proj_b, norms, embeddings and the MTP head in fp16.
~35% smaller and ~35% fewer weight bytes per token than the reference
QuantTrio/Qwen3.5-4B-AWQ (3.8 GB vs 5.7 GB), with output quality verified equivalent
in side-by-side A/B testing.
Why this exists
The reference checkpoint leaves all attention weights in fp16. Quantizing them naively
produces broken output — we root-caused two independent defects in the process:
The AutoAWQ fork's activation smoothing stores LN folds assuming a Llama-style
norm gain (w), but Qwen3.5's norm gain is (1+w) — every fold was silently wrong
at runtime while passing per-tensor checks.
Plain RTN/clipped quantization of the MLP layers loses ~3× more fidelity than the
reference build. We replaced the scale search with per-group alternating
least-squares fitting (codes → LS-refit scale/zero → re-encode, iterated), which
reproduces reference-level fidelity (rel ≈ 0.09–0.10 per module, verified against
base fp16 weights).
Measured results
QuantTrio/Qwen3.5-4B-AWQ (reference)
this model
Size
5.7 GB
3.8 GB
Single-stream decode, Radeon PRO V620 + tuned gfx1030 kernels
45.5 tok/s
97.9 tok/s (v1.1.0; 84.5 at v1.0.0)
Multi-user aggregate, 16 concurrent
—
553.1 tok/s
Multi-user aggregate, 64 concurrent
—
920.4 tok/s
Multi-user peak, 128 concurrent
—
955.8 tok/s (unchanged from v1.0.0)
IFEval (n=300, seed 42)
—
29.0 prompt-level loose acc (26.3 strict)
Quality gates: every module ≤ 0.11 relative error vs base weights (stock-dequant
reference); 5-prompt A/B against the reference checkpoint (greedy, code / history /
math / translation) — equivalent coherence, byte-identical on most prompts;
IFEval (lm-eval local-completions, n=300, seed 42) 29.0 / 26.3 prompt-level
loose/strict, 44.3 / 42.5 instruction-level.
The throughput figures above require the companion gfx1030 kernel patches
(ikantkode/gfx1030-vllm-0.26 on GitHub). On other hardware this checkpoint still
benefits any vLLM AWQ path via the reduced weight traffic.
Or one-command on a Radeon PRO V620 / gfx1030:
ikantkode/Qwen3.5-vLLM-Deploy (Docker, ~98 tok/s out of the box).
Files & lineage
Base: Qwen/Qwen3.5-4B (Apache-2.0)
Quantization: AWQ INT4, group_size 128, asymmetric zero-point, GEMM packing;
per-group scales/zeros fitted by alternating least-squares; no LN smoothing
model_mtp.safetensors: MTP head for speculative decoding (qwen3_next_mtp).
Note: on gfx1030 with heavily M=1-optimized decode kernels, MTP verification was
measurably slower than plain decoding; on other stacks it may help.
Full technical trail: ikantkode/gfx1030-vllm-0.26 (README + PROGRESS log)
Public benchmarks (lm-evaluation-harness)
Measured post-quantization, on the served checkpoint as-is — no fine-tuning,
calibration recovery, or any kind of performance restoration was performed.
Backend: live vLLM server (deployed configuration), greedy decoding, n=300/task, seed 42
Protocol note: all scores use the raw completions endpoint (no chat template).
Chat-mode vendor numbers (e.g. Qwen's IFEval 89.8) are not directly comparable —
expect a large harness/protocol gap unrelated to quantization.
Task
Shots
Metric
Score
GSM8K
5
exact_match (flexible / strict)
77.7 / 74.3
HellaSwag
0
acc / acc_norm
52.7 / 66.3
ARC-Challenge
0
acc / acc_norm
49.0 / 49.3
PIQA
0
acc / acc_norm
77.7 / 80.7
WinoGrande
0
acc
68.3
IFEval
0
prompt-level loose / strict
29.0 / 26.3 (inst-level 44.3 / 42.5)
All multiple-choice/reasoning tasks sit in the expected band for a 4B-class model;
no quantization-collapse signature. Development-time gates: per-module ≤ 0.11
rel-err vs base weights; greedy A/B vs the reference checkpoint.
Prebuilt images: ghcr.io/ikantkode/gfx1030-vllm-0.26:v1.0.0 and docker.io/exaroot/gfx1030-vllm-0.26
Revisions
v1.1.0 (Aug 19, 2026): untied lm_head quantized to INT4 (embeddings remain
fp16; tie_word_embeddings now false). All 6 gates PASS including IFEval
regression (29.0 → 29.3, sub-noise) and perplexity (+1.45%). Single-stream
84.5 → 97.9 tok/s (+15.9%); 16-user aggregate +27.6%; 128-user peak
unchanged (the head cost amortizes across the batch at high concurrency).
Docker image unchanged from v1.0.0 — only the checkpoint changed.
v1.0.0 (Aug 18, 2026): rungs 16+18 multi-user knee (673 → 966.6 tok/s
at 128 users); original full-INT4 release.
