GLM-5.2 — NVFP4 attention delta (big-3, stability-validated)
ℹ️ This is a delta, not a standalone checkpoint. It contains only the
NVFP4-quantized weights for the three largest attention projections
(o_proj, q_b_proj, kv_b_proj). Assemble a runnable model from the FP8
base + the 2-bit expert planes + this delta (see Assembly).
Update: this repo now ships the big-3 cut, which is
coherence-validated (see Quality). An earlier, more aggressive
"full attention + shared-expert" cut reached ~20 tok/s but degrades on
sustained generation (repetition / thinking-tag loops) — it is not shipped
here. Big-3 is the stable NVFP4 attention config.
Weight-only NVFP4 (4-bit) quantization of GLM-5.2's three largest attention
projections, for faster single-stream decode on 2× NVIDIA DGX Spark
(GB10, TP2).
Why this exists
Single-stream decode on unified-memory boxes (273 GB/s LPDDR5X) is
memory-bandwidth-bound — it spends its time reading weights, not
computing. After compressing the MoE experts to 2-bit, attention became the
single largest byte-read per token (~60%) while still sitting at FP8. Dropping
the biggest attention projections to 4-bit NVFP4 cuts that read.
On GB10 there is no native FP4 tensor-core MMA, so this uses weight-only
Marlin FP4: 4-bit weight read, bf16 compute — exactly the right trade for a
bandwidth-bound step.
Speed (measured, single-stream, TP2, greedy/deterministic)
| config | tok/s |
|---|
| FP8 attention (baseline) | 15.0 |
NVFP4 big-3 (o_proj,q_b_proj,kv_b_proj) — this | ~18 |
| NVFP4 full attention + shared (aggressive; degrades) | ~20 |
~+20% over the FP8-attention baseline, at coherent quality. Weights
quantized at ~0.09 mean relative L1 error. Real-world sampled throughput
varies with MTP speculative-decode acceptance; greedy shows the clean number.
Quality
Coherence-validated (2× Spark, TP2, MTP on): greedy 320-token generations
stay coherent to the end, and short reasoning checks are correct (e.g. the
"all-but-9 sheep" riddle → 9; a two-step word problem → correct total). This is
the config that stays stable where the full-attention cut collapses.
Full task battery (GPQA / GSM8K / IFEval / MMLU-Pro) still pending — coherence
is not the same as task parity with the FP8 / 2-bit baselines. Do not assume
benchmark parity yet; check back or open a discussion.
What's in this repo
nvfp4-dense-000{0..3}.safetensors — NVFP4 weights for o_proj, q_b_proj,
kv_b_proj (weight uint8 packed FP4, weight_scale fp8-e4m3 per-16 group,
weight_scale_2 fp32 global = amax/2688).model.safetensors.index.json — the overlay index (these three projections →
the NVFP4 shards; everything else → the FP8 base).config.json — GLM-5.2 config (fp8 quantization_config; NVFP4 linears are
selected at serve time by the loader hook + VLLM_NVFP4_TARGETS).
Assembly (this is a delta)
Runnable stack = FP8 base + 2-bit expert planes + this NVFP4 delta + the
vLLM-Moet NVFP4 patches:
- FP8 base:
zai-org/GLM-5.2-FP8
- 2-bit pruned expert planes:
sapidlabs/GLM-5.2-2bit-MoE-planes-pruned208-tp2
- This NVFP4 attention delta (overlay dir over the FP8 base).
- Code:
Sapid-Labs/vLLM-Moet,
branch spark-gb10 — see spark/NVFP4-DENSE.md for the packer
(spark/prepack_nvfp4_linear.py), the env-gated loader hook
(VLLM_NVFP4_DENSE=1), and the serve command.
Serve (2× Spark, TP2), pointing the model at the overlay dir — note the
big-3 target list:
1MODEL=<overlay_dir> VLLM_MOE_W2_PREPACKED_DIR=<planes_dir> \
2VLLM_NVFP4_DENSE=1 \
3VLLM_NVFP4_TARGETS="o_proj,q_b_proj,kv_b_proj" \
4MTP_K=1 VLLM_ENGINE_READY_TIMEOUT_S=2400 \
5bash spark/serve-glm52-tp2-mtp.sh
License
MIT, following the base model zai-org/GLM-5.2-FP8.
