GLM-4.5-Air — Route-Weighted EXL3, 2.25 bits/weight
TL;DR: a 106B-parameter MoE compressed to ~30 GB that runs at ~90 tok/s on a single
RTX 5090 — quantized with route-weighted per-expert Hessians (gate²-weighted
curvature, the first-order-correct objective on the MoE graph) instead of the standard
unweighted calibration. At identical bits, it measures better than the stock EXL3
quantization on every aggregate metric.
Measured head-to-head (identical rate, identical eval protocol, single RTX 5090)
| metric | this model | stock EXL3 2.25bpw |
|---|
| KL vs BF16 teacher (wiki2, 40×2k) | 0.2571 (−11.3%) | 0.2900 |
| Top-1 token agreement w/ teacher | 82.8% | 81.3% |
| MMLU (600q, 5-shot) | 73.2% | 72.8% |
| HumanEval pass@1 (greedy) | 78.0% | 75.6% |
| Decode / prefill tok/s | 90.3 / 2,429 | 88.0 / 2,001 |
| Rare-knowledge screen (205 probes) | 146/205 | 150/205 (Δ within noise: McNemar p≈0.65) |
Replication on a second architecture (Qwen3.6-35B-A3B, same recipe, same protocol):
route-weighted KL 0.1026 / top-1 87.2% vs stock-quantized 0.1076 / 86.6% — same direction,
smaller margin (−4.6%).
Honest limitations
- 2.25 bpw is aggressive: KL vs the BF16 teacher is ~0.26 nats — clearly better than the
alternative at this size, not indistinguishable from the original. If you have the
memory, use more bits.
- Our certification includes a tail-knowledge deletion screen (rare facts die first
under compression — standard benchmarks don't see it). Results for this artifact are
reported above and in the paper, including anything unflattering.
- EXL3 format: requires exllamav3 (CUDA). Not a GGUF; llama.cpp/ollama not supported.
Method (short version)
In a mixture-of-experts model, each expert's contribution to the output is scaled by its
router gate. The chain rule therefore says expert weights should be quantized under a
curvature weighted by gate², accumulated only over tokens actually routed to that
expert, with projection-specific statistics — not under the usual all-tokens,
unweighted proxy. We apply that curvature inside a trellis-coded quantizer (QTIP-family)
via LDLQ. A ~45-line patch to ExLlamaV3's converter implements it; the patch,
certification harness, and probe set are public (method repository link in the repo discussions — posted alongside the release). Full derivation,
controls (data-free and rate-matched-scalar), scale studies, and negative results are in
the paper: [PAPER-LINK].
Reproduce the evaluation
All numbers above are reproducible with the public harness against this repo and
turboderp/GLM-4.5-Air-exl3 (2.25bpw branch) as the baseline.
Credit
Base model: GLM-4.5-Air (Z.ai, MIT). Runtime/quantizer: ExLlamaV3 (turboderp).
Trellis codebook lineage: QTIP (Cornell RelaxML), Q-Palette. Certification philosophy:
statistically-lossless framework (IST Austria). Mistakes are ours alone.
