Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa).
This is the quality-flagship sibling of GLM-5.2-Alis-MLX-Dynamic-3.5bpw: the routed experts (≈96% of all parameters) are promoted from 3-bit affine to NVFP4 (4-bit float, group-size 16) — a ~2.2× reduction in expert weight-space relative error (rel-err) — while everything else (recipe, MLA/DSA fixes, int8 MLA-KV, native MTP layer) is identical. Choose by context budget on a 512 GB M3 Ultra:
this build (4.5 bpw, nvfp4 experts)
3.5 bpw sibling
quality
higher (expert rel-err 0.104 vs 0.225)
good
max context in 512 GB (int8 KV)
~500–600K tokens
~1M tokens
decode, tok/s (single request, short ctx)
16.3 · 18.7–19.0 with --mtp
20.4 · 20.7
Native MTP included: the checkpoint ships GLM-5.2's MTP (nextn) layer for lossless self-speculative decoding (--mtp). On this cleaner 4-bit target, speculation pays measurably more than on the 3-bit sibling — accept-len 2.61–2.87 and +15–16% (vs ~2.3 and ≈+2% on the sibling) — see Native MTP.
⚠️ Requires a patched mlx-lm with the glm_moe_dsa correctness fixes and int8 MLA-KV support — and, new for this build, an mlx core with NVFP4 kernels: mlx ≥ 0.30.3 (tested on 0.31.2). Older runtimes fail to load mode="nvfp4" tensors.
This build uses IndexShare — the DSA indexer is stored only on the 21 "full" layers ([0,1,2,6,10,…,74]); the other 57 layers reuse it. Stock mlx-lm / omlx build an indexer on every layer, so loading fails with ValueError: Missing 285 parameters: …self_attn.indexer… (57 shared layers × 5 params). Use avlp12/mlx-lm, branch glm-5.2-dsa-inference (upstream PR ml-explore/mlx-lm#1419, open as of writing), which adds IndexShare + non-interleaved indexer RoPE + int8 MLA-KV. Serve with mlx_lm.server from that fork. On stock mlx-lm, --kv-bits is also a silent no-op for this architecture (see Correctness and Long context).
The routed experts are ~96% of all parameters; their precision dominates build quality. At the same storage cost as 4-bit affine, NVFP4 (E2M1 values + FP8-E4M3 per-16 group scales + per-tensor FP32 global scale) cuts expert reconstruction error by ~7% on heavy-tailed expert weights vs affine-4, and by ~2.2× vs the 3-bit sibling:
Per-layer conversion telemetry across all 75 MoE layers (gate/up/down): rel-err 0.1006–0.1035, no outlier layers (threshold 0.13).
End-to-end PPL (strided, ctx 2048 / stride 1024)
this build
3.5 bpw sibling
Δ
wikitext-2 (prose)
2.713
2.946
−7.9%
code (mlx_lm src)
1.821
1.893
−3.8%
Strided perplexity from a fixed local harness — relative numbers for comparing builds, not directly comparable to perplexities other quantizers report on different corpora.
~455–460 GB estimated — brushes the ~460 GB wired limit
Format
MLX (Apple Silicon), requires mlx ≥ 0.30.3
Context
~500K comfortable / ~600K max in 512 GB (int8 KV; DSA sparse attention) — for 1M use the 3.5 bpw sibling
Speculative decoding
native MTP (nextn layer 78 included)
Recipe
Identical to the 3.5 bpw sibling except the expert tier — bits are allocated by sensitivity: cheap-but-accurate float4 on the robust expert bulk, full precision on the discrete routing/sparse-attention control paths.
Component
Bits
Why
Routed experts (gate/up/down)
NVFP4 g16 (4-bit float)
~96% of params — the bulk; float format wins on heavy-tailed experts
The NVFP4 expert tensors are stored as per-projection stacked 3-D weights (switch_mlp.{gate,up,down}_proj) with {"group_size": 16, "bits": 4, "mode": "nvfp4"} per-path overrides in config.json (225 entries, 75 MoE layers × 3 projections). gather_qmm with mode="nvfp4" was gate-checked for exactness (measured max-abs difference vs the dequantized-weight reference: 0.000000 on the expert forward) and speed (parity with same-bpw affine at decode batch sizes, ±1%).
Long context (~500–600K) — int8 KV
GLM-5.2's MLA stores the compressed latent (kv_lora 512 + rope 64 per layer), so the KV cache is small: ~95 GB per 1M tokens at fp16, ~48 GB per 1M at int8. At this build's weight footprint the practical ceiling on 512 GB is ~500K tokens comfortable, ~600K max with int8 KV (at 600K the estimated peak, ~455–460 GB, brushes the ~460 GB wired-memory limit); for a full 1M-token budget use the 3.5 bpw sibling.
The patched runtime quantizes only the MLA latent cache to int8 (the DSA indexer cache stays fp16) and dequantizes the latent on read inside MLA attention. At ≤128K context, fp16 KV is fine (a few GB); int8 KV extends the ceiling from ~350K (fp16) to ~500–600K.
Why int8 and not fp8? Measured on the actual latent (post-kv_a_layernorm, std ≈ 0.006, max ≈ 0.03): int8-affine (g64) cos 0.99998 vs MXFP8 (g32) 0.9997 vs plain fp8 e4m3 0.994 — int8 is 100–200× lower MSE on both the latent and the decoupled RoPE key. fp8's wider dynamic range doesn't help a narrow, well-conditioned distribution where precision (mantissa bits) is the binding constraint, and MLX dequantizes the latent to fp16 for the MLA matmul anyway. (Note this is about the KV cache; the expert weights are a different, heavy-tailed distribution — that's where a float4 format does win, hence nvfp4 experts + int8 KV in the same build.)
Native MTP — self-speculative decoding
GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. This build includes it as model.layers.78.* (one extra shard, +4.5 GB). The MTP shard is reused unchanged from the 3.5 bpw sibling (experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16) — drafter precision was measured not to affect acceptance (bf16 ≡ 3-bit), so the drafter is deliberately kept small while the nvfp4 promotion applies to the backbone experts only.
Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave identically to a non-MTP checkpoint. With the fork, --mtp turns it on:
MTP pays more on this build than on the 3-bit sibling. Speculation gains scale with target precision — a cleaner target accepts more drafts (its own outputs are less noisy) — measured across this quant family: 3-bit experts ≈ +2% (accept-len ~2.3, the 3.5 bpw sibling), 4-bit-class experts ≈ +11% (accept-len ~2.46, measured on a 4-bit sibling). Measured on this build (M3 Ultra 512 GB, single request, greedy, short context):
plain
--mtp (k=2)
--mtp (k=3)
decode
16.34 tok/s
18.71 tok/s (+15%)
18.95 tok/s (+16%)
mean accepted length
—
2.61
2.87
It stays exactly lossless — at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact) — so enabling it costs nothing but the ~4.5 GB layer.
Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime); the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration. Measured on this build at ~2.15K context, prefill-separated: plain 15.89 tok/s, --mtpk=1 ≈ 0.98× (accept-len 1.79), k=2 ≈ 0.86× (accept-len 2.31), k=3 ≈ 0.79×. Guidance: k=2–3 in the ≤2048 dense regime; at longer contexts prefer --mtp-num-draft-tokens 1 or plain — the residual gap is acceptance economics (accepted length vs the ~2.1+ break-even in the sparse regime), not path cost.
Notes for other integrators (the three footguns, all verified against vLLM's deepseek_mtp semantics):
eh_proj input concat order is [enorm(embed(t+1)), hnorm(h_t)] — embedding first.
The hidden fed to the MTP is the backbone's pre-final-norm residual stream.
For chained multi-token drafting, feed back the MTP's shared_head.norm-normalized hidden — chaining the raw layer output halves chained acceptance (measured 0.27 → 0.75 conditional at position 2).
How it compares
Within this two-build family, the 3.5 bpw sibling is the 1M-context build; this one is the quality build (~500–600K ctx). Against other public GLM-5.2 MLX builds in the 4.5–5 bpw tier, this one is the smallest on disk, and it is the only one of the compared builds shipping the native MTP layer, int8 MLA-KV, and the DSA correctness fixes:
this build
3.5 bpw sibling
Q4.8-INF
DQ4plus-q8
effective bpw
~4.5 (experts nvfp4)
~3.5
~4.8
~5.0
on-disk
424 GB
333 GB
447 GB
465 GB
native MTP layer
✓ included
✓ included
✗ stripped
✗ stripped
DSA indexer
fp16
fp16
(custom fmt)
8-bit
int8 MLA-KV
yes
yes
no
no
max ctx in 512 GB
~500–600K (int8 KV)
~1M (int8 KV)
fp16 KV only — 1M ✗
fp16 KV only — 1M ✗
Per-component bit allocation was parsed from each build's config.json. The nvfp4 expert tier gives this build higher weight fidelity than Q4.8-INF/DQ4plus on the dominant parameter mass at 20–40 GB less footprint, plus the runtime correctness fixes the others lack.
Benchmarks
mlx_lm.perplexity (seq 2048, 50 samples, seed 123) — the same harness and settings as the sibling cards, against the author's earlier builds:
Note (2026-07-10): the GLM-5.1 · 2.7 bpw build referenced below (avlp12/GLM-5.1-Alis-MLX-Dynamic-2.7bpw) has been removed from the Hub to reclaim storage; its figures are kept here for reference only.
GLM-5.1 · 2.7 bpw
GLM-5.2 · 2.56 bpw
GLM-5.2 · 3.5 bpw
GLM-5.2 · 4.5 bpw (this)
Perplexity (lower)
4.165
3.850
3.766
3.812 ± 0.031
On this small-sample chat corpus (allenai/tulu-3-sft-mixture, the mlx_lm.perplexity default) the two 512 GB-class siblings are within noise of each other; the higher-precision strided measurement in Quality above (−7.9% wikitext / −3.8% code, ~23–28K scored tokens per corpus) is the discriminating comparison. A 0-shot task-accuracy table (HellaSwag / PIQA / WinoGrande) may be added in a future card update.
Correctness (verified vs the HF reference)
glm_moe_dsa needs fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:
IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
Indexer RoPE / eps — non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Baked into config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6); post-RoPE q matches the reference to ~1e-7.
int8 MLA-KV — CacheList.to_quantized/offset + MLA dequant-on-read, so --kv-bits 8 actually engages for the latent cache (silently ignored on stock mlx-lm).
Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding, chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).
NVFP4 expert path — gather_qmm(mode="nvfp4") output matches the dequantized-weight reference exactly on the expert forward, at affine-parity speed; mixed nvfp4+affine per-layer config round-trips through save → load_model → generate (gate-checked before the build).
Validation (this build): full load under the wired-limit protocol (peak 424 GB, ~2.15K-ctx run); coherent greedy generation at short and ~2.15K context; MTP speculative path exercised at k=1–3 (accept-len 2.61–2.87 short / 1.79–2.55 long — greedy-identity is enforced by the acceptance rule, verified exhaustively on the sibling); per-layer conversion telemetry rel-err 0.1006–0.1035 across all 75 MoE layers (no outliers); config round-trip with 225 nvfp4 per-path overrides. Strided PPL vs the sibling in Quality above. Fork validated at branch head 2b2138f.
Usage
bash
1# requires the fork (glm_moe_dsa fixes + int8 MLA-KV + --mtp) on mlx ≥ 0.30.32mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \3 --prompt "Write a quicksort in Python."
Hardware
Built for 512 GB Apple Silicon (M3 Ultra). Weights ~424 GB (395 GiB); with int8 KV a ~500K-token context runs comfortably in 512 GB (~600K max). For a full 1M-token context, or for more free headroom, use the 3.5 bpw sibling; for ≤256 GB machines, the 2.56 bpw build.
Credits
Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
MLX & mlx-lm: Apple ml-explore (NVFP4 kernels since mlx 0.30.3).
