[!IMPORTANT]
Requires llama.cpp b10251 or newer. These files keep the MTP (nextn) draft head, which
older builds do not know how to create, so they refuse to load the file:
error loading model: done_getting_tensors: wrong number of tensors; expected 868, got 862
That is a loader version problem, not a corrupt download — the file has all 868 tensors, but
a pre-b10251 build only asks for 862 because it has no GLM-4.7-Flash MTP support
(#24868, commit 57c0921). Update
llama.cpp and it loads. Open an issue here if you need an MTP-stripped build for an older
runtime.
MoE-aware, mixed-precision APEX quantization of
zai-org/GLM-4.7-Flash — 31.2B total /
~3.6B active, and the most-downloaded model in the GLM family by a wide margin.
Three tiers, all built from the bf16 checkpoint (no intermediate requantization), all
imatrix-guided, all with the MTP draft head preserved and quantized so speculative
decoding works out of the box.
Architecture, and why it quantizes well
Despite the GLM name this is a DeepSeek-lineage design — llama.cpp converts it via
Glm4MoeLiteModel(DeepseekV2Model) to the deepseek2 GGUF architecture.
1 nextn layer — a full MoE block + draft projections, 1.28B params
Two properties make it an unusually good APEX target:
Routed experts are 89% of all parameters (27.8B of 31.2B). Everything whose error
compounds on every token — MLA attention, shared experts, embeddings — is only ~2.1B, so
all of it stays at ~6.5–6.9 bpw for about 1.8GB while the entire remaining budget goes to
experts.
moe_intermediate_size is 1536, cleanly divisible by 256, so all three expert
matrices (gate/up/down) can use the full k-quant and IQ menu. Models with a non-256
expert dimension cannot: their ffn_down_exps is restricted to block-32/64/128 types.
MLA keeps long context affordable. Instead of caching full K and V per head, it caches
one 576-dim latent per token per layer:
KV cache @ 198K ctx, bf16
size
MLA (512+64 latent)
11 GB
the same model as plain MHA
342 GB
Tiers
tier
file size
bits/weight
routed-expert bpw
non-expert bpw
wikitext PPL
vs bf16
bf16 (reference)
62.5 GB
16.00
16.00
16.00
12.112
—
APEX-i-quality
16.60 GB
4.25
4.03
6.92
12.746
+5.2%
APEX-i-compact
13.27 GB
3.40
3.05
6.65
13.837
+14.2%
APEX-i-mini
10.24 GB
2.62
2.24
6.39
15.801
+30.5%
Allocation is by tensor role and layer band: edge layers keep more bits than the redundant
middle, routed experts absorb the compression, and shared experts / MLA projections /
embeddings stay high. The MTP draft head is pinned to Q4_K in every tier (see below).
On the PPL column: the imatrix is deliberately code-weighted (~70% multi-language code,
30% prose) and these numbers are measured on wikitext, so they are the pessimistic direction
of that domain mismatch. The task benchmark below tracks quality much better — i-quality
gives up 5.2% PPL but only 3.6 points of task score.
Agentic coding benchmark
Five hard from-scratch and fix-existing-code challenges driven through a real agentic loop
(OpenCode), scored by partial credit over independently re-run pytest — never the agent's
own self-report. MTP speculative decoding enabled (n_max=2). One run per tier.
challenge
bf16
i-quality
i-compact
i-mini
batch_scheduler
10/10
10/10
10/10
10/10
buddy_allocator
7/11
5/11
5/11
5/11
circuit_breaker
7/7
7/7
7/7
5/7
expr_parser_advanced
20/20
20/20
20/20
7/20
inventory_fix_and_extend
11/11
11/11
11/11
11/11
test cases
55/59 (92.7%)
53/59 (89.1%)
53/59 (89.1%)
38/59 (70.4%)
challenges fully passed
4/5
4/5
4/5
2/5
wall clock
21.3 min
8.9 min
13.1 min
11.5 min
This is a sweep, not a cutoff claim. Three sizes are shipped with their measured degradation
so you can pick against your own budget — one model, one workload, one run per tier is not
enough to assert a general "quantize no further than X bpw" threshold, and none is asserted
here.
Read honestly:
i-quality and i-compact are indistinguishable (both 53/59) despite a 24% difference in
expert precision and 3.3GB of file size. Between 4.03 and 3.05 bpw the compression is
effectively free on this workload.
i-mini is genuinely weaker — 70.4%. It holds on the shorter challenges and degrades on
the longest, most compositional one (expr_parser_advanced, 20/20 → 7/20). This is the one
tier where PPL and task score agree that something real was lost. It is published because a
10.2GB file competes against other ~10GB models, not against the larger tiers here — but
that comparison is not benchmarked, so judge it on its own numbers.
bf16's 2-test-case lead is a single run on one challenge and is within plausible
run-to-run variance for a stochastic agentic loop. Treat it as "bf16 is not clearly better
than i-quality" rather than a measured 3.6-point gap.
Wall clock mixes decode speed with how long each tier flailed on the hard challenge. The
cleanest same-work comparison is expr_parser_advanced, identical 20/20: bf16 404s vs
i-quality 157s (2.6×).
MTP speculative decoding
The nextn draft head is preserved and quantized (not stripped). llama-imatrix never
exercises it — a plain forward pass reports the whole MTP layer as unused — so it receives no
imatrix data, and is pinned to Q4_K rather than an IQ type, which would be fitted against
statistics that do not exist for it.
Measured on i-mini:
decode
without MTP
78.7 t/s
with MTP (n_max=2)
116.2 t/s (+48%)
Draft acceptance across tiers, from the benchmark runs:
tier
drafter
acceptance
accepted / verify step
bf16
bf16
58.5%
—
i-quality
Q4_K
65.2%
—
i-compact
Q4_K
59.0%
—
i-mini
Q4_K
64.5%
1.29
Acceptance is ~59–65% for every quantized tier, from 4.03 down to 2.24 bpw experts — the
speculative speedup survives aggressive quantization intact. Notably the bf16 file, whose
draft head is also bf16, posts the lowest acceptance of the four: a low-precision drafter is
not a compromise here, which matches independent results on other MoE checkpoints.
1# chat / agentic serving (add --spec-type draft-mtp for the MTP speedup)2llama-server -m GLM-4.7-Flash-APEX-i-quality.gguf --jinja -ngl 999 --ctx-size 3276834# one-shot5llama-cli -m GLM-4.7-Flash-APEX-i-quality.gguf --jinja -ngl 999 --ctx-size 8192 -st \6 -p "Write a Python LFU cache with O(1) get and put."
This is a reasoning model — it emits thinking before its answer, so allow a generous token
budget or responses will be truncated mid-thought. For multi-turn agentic use, ZAI recommends
Preserved Thinking mode; llama.cpp exposes --reasoning-preserve.
Requires llama.cpp b10251 or newer (see the note at the top): Glm4MoeLite conversion
support plus GLM-4.7-Flash MTP support (upstream #24868).
Reproducing
Converted from the bf16 checkpoint with convert_hf_to_gguf.py (no intermediate
requantization).
imatrix: llama-imatrix, ctx 512, over a ~70% multi-language code / 30% prose corpus,
published at
Myric/Laguna-S-2.1-imatrix-calibration-study
(corpora/laguna_calib_codeweighted.txt). Expert coverage 100% except 9 expert stacks at
98.44% (63 of 64 experts observed).
Per-tensor allocation emitted as a --tensor-type-file consumed by stock llama-quantize;
no kernel or C++ changes.
PPL: llama-perplexity on wikitext-2 test, ctx 512.
