GLM-5.2-REAP-NU176-526B — non-uniformly expert-pruned GLM-5.2 (NVFP4)
The first non-uniform REAP cut of GLM-5.2: instead of pruning every MoE layer to the
same expert count, each layer keeps a different number of routed experts (108–216,
average 176 of 256) allocated by per-layer REAP saliency measured over 16.08B
calibration tokens. ~526B params, 337GB on disk, NVFP4 routed experts.
Why non-uniform
Uniform keep-K pruning drops 16–26% of per-layer routing mass depending on the layer —
layers 7–17 of GLM-5.2 are hurt far more than layers 3–6 or 77. This cut reallocates the
expert budget to where the router actually concentrates saliency:
| layer range | keep (of 256) |
|---|
| 3–6 | 108–168 |
| 7–17 | 176–216 (peak need) |
| 18–48 | 140–176 |
| 49–77 | 136–184 |
| 78 (MTP) | 168 (NVFP4) |
Mean dropped saliency mass: 17.6% vs 22.3% for the prior uniform-156 cut.
⚠️ Requires a (tiny) vLLM fork
Stock vLLM assumes one expert count for all layers. This checkpoint carries
config.num_routed_experts_per_layer (list of 79 ints; n_routed_experts = max = 216)
and each layer's experts are renumbered 0..k−1 with matching gate.weight /
e_score_correction_bias rows.
The required patch is 12 lines in deepseek_v2.py (DeepseekV2MoE resolves its
layer's count from the list; everything else — weight loading, MTP, ModelOpt NVFP4,
CUDA graphs, TP, DCP — works unmodified because the scalar stays at the max).
One-command serving (docker, 4× RTX PRO 6000 Blackwell SM120 or similar 4×96GB):
see
https://github.com/0xSero/glm-5.2-nonuniform — 262k context @ concurrency 2–3
(DCP4, FP8 KV, MTP speculative decoding; ~40 tok/s at DCP4/262k, ~77 tok/s at DCP2/131k).
Prebuilt forked-vLLM image (pick either):
- GHCR:
docker pull ghcr.io/0xsero/vllm-b12x-nonuniform:20260705
- This repo:
docker/vllm-b12x-nonuniform-20260705.docker.tar.zst
(10.6GB) — zstd -dc *.docker.tar.zst | docker load
How it was made
- Per-(layer, expert) REAP saliency (gate-weight × expert-output norm) merged from 5
calibration runs (GLM-5.2 + GLM-5/5.1 cross-checks), 16.08B tokens.
- Per-layer keep counts via water-filling on layer-normalized saliency curves
(budget = 75×176; counts clamped to [108, 216], multiples of 4).
- Whole-expert copy from
nvidia/GLM-5.2-NVFP4 with per-layer renumbering, router rows
sliced; MTP layer experts taken from the NVFP4 uniform cut (NVIDIA ships MTP in BF16,
which b12x-class MoE kernels reject).
- Anti-overthinking serving per arXiv:2606.00206 (quantized/pruned reasoning models
overthink; a hesitation-marker penalty/directive restores CoT efficiency).
Observation data: 0xSero/glm-5.2-reap-observations, 0xSero/glm52-reap-traces.
Lineage
- Base: nvidia/GLM-5.2-NVFP4 (753B-A40B, 78L, 256E, MLA+DSA, MTP)
- Method: REAP (Router-weighted Expert Activation Pruning), extended to per-layer budgets
- Prior uniform cut:
0xSero/GLM-5.2-504B (keep-168)
