GLM-5.2-504B (REAP keep-168, NVFP4)
A 34%-pruned GLM-5.2 — 168 of 256 routed experts kept per layer (incl. the MTP layer),
NVFP4-quantized, ~504B params, recovered via gate-only Router-KD to the unpruned teacher.
This is the largest / highest-quality of the sibling cuts, and on a well-powered real-world eval it
reaches parity with the full unpruned GLM-5.2.
🙏 Sponsor
Pruning, distillation, and evaluation ran on
8× NVIDIA B200 sponsored by Lambda.
Thank you, Lambda. 🙏
What this is
- Arch:
GlmMoeDsaForCausalLM — 78 layers (3 dense + 75 MoE) + 1 MTP layer, DeepSeek Sparse
Attention, sigmoid router (top-8), 1 shared expert, hidden 6144.
- Prune: REAP (saliency =
gate × ‖expert_output‖) → top-168/layer, consistent across all MoE
layers and the MTP layer; n_routed_experts: 168 (loads cleanly in vLLM).
- Quant: NVFP4 (modelopt) routed experts; BF16 router / attention / shared expert.
Recovery (Router-KD)
Freeze experts + backbone; train only the 75 router gates (~0.016% of params) to KL-match the
unpruned GLM-5.2 teacher's next-token distribution (plain uniform weighting, lr 5e-5).
Eval — n=2000 held-out real prompts (raw sampling, no max_tokens / no timeout)
Loops are detected, not truncated. 2000 probes harvested from real coding-agent traces (codex,
opencode, cursor, claude-code), held out from training.
| metric | keep-168 + Router-KD |
|---|
| attractor / loop rate | 0.072 |
| natural-EOS rate | 0.928 |
| output diversity (distinct-4) | 0.880 |
| median output length | 1267 tok |
At this scale the difference vs the unpruned teacher is within noise — i.e. parity, measured on
2000 samples (not the usual n=50). The residual loops are inherent to GLM-5.2 (the unpruned teacher
exhibits the same </think>-restart loops on the same prompts), so they're not a pruning artifact.
Serving (vLLM)
1vllm serve 0xSero/GLM-5.2-504B --tensor-parallel-size 8 \
2 --quantization modelopt_fp4 --kv-cache-dtype fp8 --trust-remote-code --max-model-len 262144
Tip — brevity prompt: a system prompt like "Be concise. Think only as much as the task needs,
then answer and stop." roughly halves median output length at no retraining cost.
More
- GGUF builds (BF16 + Q4/Q3/Q2 dynamic):
0xSero/GLM-5.2-REAP-504B-GGUF
- Siblings:
0xSero/GLM-5.2-481B (keep-160), 0xSero/GLM-5.2-469B (keep-156)
Compute sponsored by Lambda — thank you. 🙏
Honest note (n=2000)
The unpruned teacher loops on only 3.6% of these prompts vs ~7-8% for this pruned cut — REAP pruning roughly doubles the loop rate, and gate-only Router-KD (even on full data) does not close it. Earlier small-n evals suggesting parity were a sampling fluke. A knowledge-recovery LoRA is in progress to add capacity back.
