SmolLM3-3B quantized with
GLQ using the QTIP-style trellis (TCQ) codebook at
6 bits/weight, via stacked residual vector quantization: a K=4 primary trellis plus a
K=2 residual stage.
[!IMPORTANT]
Requires glq >= 0.8.0 — that release added stacked residual VQ end to end: the
loader, the CUDA decode kernel and vLLM serving. Older builds decode the primary stage
only, silently giving 4 bpw quality.
Quality
Wikitext-2 test perplexity, seqlen 2048, 141 windows, measured through one harness on an
RTX PRO 6000 Blackwell:
PPL
vs bf16
bf16
9.1220
—
this checkpoint (6 bpw)
9.1310
+0.10%
GLQ trellis 4 bpw
9.2299
+1.18%
6 bpw closes ~92% of the 4 bpw → bf16 perplexity gap. Scope these numbers to this run:
wikitext-2 perplexity on a 3B model is a single-metric quality signal and does not by itself
establish downstream task parity. No reasoning or knowledge benchmarks have been published
for this checkpoint yet.
On disk: 2516.55 MiB of safetensors, which back-solves to exactly 6.00 bits per quantized
weight (the unquantized remainder is the tied embedding).
Serving is supported as of 0.8.0; the GLQ plugin registers itself through
vllm.general_plugins, so no extra flags beyond the quantization name are needed.
Measured on an L40S (sm_89) with vLLM 0.25.1, CUDA graphs captured:
weights in VRAM
decode B=1
decode B=32
6 bpw (this checkpoint)
2.46 GiB
89.7 tok/s
1076.7 tok/s
GLQ trellis 4 bpw
1.81 GiB
131.0 tok/s
1768.9 tok/s
vLLM and transformers-eager produce identical greedy output, token for token (64/64) on
this checkpoint, and greedy output at B=32 is identical across all 32 sequences and to B=1.
Numbers are specific to this GPU and vLLM version — a bandwidth-starved card and a
96 GB-class card do not scale the same way.
The checkpoint records variant: 3inst in its config, so the decoder selects itself; the
GLQ_TRELLIS_VARIANT environment variable is a quantization-time setting only.
Decoding a stacked-RVQ checkpoint costs about 1.9× the 4 bpw kernel at batch 1 in
isolation, which shows up as roughly 1.46× end to end under vLLM at B=1 (the kernel is
not the whole decode step). That cost is architectural rather than a missing optimization:
trellis decode cost is per trellis-state rather than per bit, so two stages decode twice as
many states regardless of how they are scheduled. Prefill is affected far less (~1.21×),
because it runs the dense branch.
