GLM-5.2 — mixed-bit VQ (AQLM) ~1.90 bpw

What it is

• Experts: mixed 1 / 2 / 3-bit vector quantization (AQLM-style — per-group codes + shared codebook + GPTQ error compensation). Module bit distribution {1: 29456, 2: 12441, 3: 15703}.
• Non-expert spine (attention / router / norm / embed / lm_head): scalar 4/8-bit.
• Size: ~172 GiB. Allocation: loss-aware (output-Fisher × input-energy × quantization error, an Optimal-Brain-Quantization 2nd-order cost) at a 1.90 bpw budget.

Quality (KL to BF16, lower = better; fake-quant on held-out corpora)

Serving (sm_120 / consumer Blackwell)

• Backend: vLLM + OneCompression VQ kernels + an sm_120 sparse-attention gather fallback (sm_120 ships no FlashMLA sparse kernel). Tensor-parallel = 2. See vqmoe.
• Context: 16384 (sparse). KV at gpu_util 0.97 ≈ 25,856 tokens.
• Weights: ~87.3 GiB/GPU. Throughput: prefill ~15× vs dense; decode ~13–16 tok/s.
• Defaults: no-think (answers in content; thinking is opt-in per request via chat_template_kwargs {"enable_thinking": true}), with LZ-penalty + 3000-token think-budget forcing as low-bit reasoning safety nets.

Limitations

• Thinking mode does not always self-terminate on casual prompts at this bit-width (an inherent sub-2-bit "no-exit" attractor); the no-think default + budget-forcing contain it.
• Decode ~16 tok/s is an sm_120 MLA single-query ceiling, not tunable here.
• Context > 16k and multi-query speculative decode are unavailable on sm_120 with current kernels.
• Reproducing the exact serving stack additionally needs the sm_120 vLLM patch stack — see the vqmoe repo's "Reproducibility status".

License