Gemma 4 26B-A4B · REAP-25 · Q3_K / Q2_K mixed (imatrix) — GGUF
A
9–10 GB mixture-of-experts specialist derived from
google/gemma-4-26B-A4B-it by
pruning 25 % of routed experts with routing-aware REAP and quantizing the
result with attention-aware mixed-precision schemes calibrated by
llama-imatrix.
Two quantizations are provided in this repo:
| File | Size | HumanEval | MBPP | When to pick |
|---|
Gemma4-26B-A4B-REAP-25-Q3_K-mixed.gguf | 10.3 GB | 92.7 % | 70.4 % | Default. Best quality/size trade |
Gemma4-26B-A4B-REAP-25-Q2_K-mixed.gguf | 9.1 GB | 90.9 % | — | Tightest memory (headroom for KV / other apps) |
The Q4_K_M (13 GB) baseline scored 91.5 % HumanEval — bigger AND lower quality
than the Q3_K mixed variant, so it isn't shipped here.
TL;DR
| Base · bf16 | REAP-25 · bf16 | REAP-25 · Q3_K-mixed (this repo) |
|---|
| Parameters | ~26 B | ~19.5 B | ~19.5 B |
| File size on disk | ~52 GB | ~40 GB | 10.3 GB |
| HumanEval | 93.9 % | 93.9 % | 92.7 % |
| MBPP | 73.0 % | — | 70.4 % |
| IFEval (prompt / instr) | — | — | 80.4 % / 86.6 % |
| MMLU-Pro (5-shot) | 82.6 %* | — | 76.6 % |
* MMLU-Pro base score is from the Gemma 4 published model card, not
re-measured here. All other numbers are from this project's own harness.
Method
Compression: routing-aware REAP, 25 % expert pruning
Routed experts are scored by router-weighted activation magnitude and a
routing-disruption penalty (the routing-aware variant of REAP); the lowest-
scoring 25 % per layer are pruned. See arXiv:2510.13999 for the method.
Why routing-aware here (and not on Qwen 3.6): Gemma 4 has no shared expert
to absorb load when a routed expert is pruned, so the routing-disruption
penalty measures real quality cost rather than being double-counted by the
architecture. On models with a shared expert (Qwen 3.6, DeepSeek, etc.), plain
REAP without the routing term is sufficient.
Result before quantization: ~26 B → ~19.5 B parameters; bf16 size ~40 GB.
Quantization: mixed profiles + imatrix
Both files use llama.cpp's Q3_K_S selection base with attention overridden
to Q8_0, calibrated with llama-imatrix on the same code / tool-use / math
mix used for REAP scoring. Approximate distribution (658 tensors total):
| Tensor group | Q3_K-mixed file | Q2_K-mixed file |
|---|
| Attention (attn_q/k/v/output), embeddings, output | Q8_0 (129 tensors) | Q8_0 (129 tensors) |
| Mid-precision body | Q4_K + Q5_0 + Q4_0 (106) | Q4_K + Q5_0 + Q4_0 (106) |
| Lowest-precision body | Q3_K (30) | Q2_K (30) |
| Layer norms / small | F32 (392) | F32 (392) |
The two files differ only in those 30 lowest-precision tensors — Q3_K vs Q2_K
— which is where the ~1 GB size difference comes from.
Inference
llama.cpp / llama-server (recommended)
1llama-server \
2 -m Gemma4-26B-A4B-REAP-25-Q3_K-mixed.gguf \
3 --port 8093 --host 0.0.0.0 \
4 -c 8192 -ngl 999 \
5 --jinja --reasoning off
Recommended sampling
temperature: 0 (deterministic) or 0.2 for variationmax_tokens: 2048 for code, higher for long-form
Performance
Not re-measured here for the two shipping quants specifically. Reference
numbers from the same-family Q2_K mixed on M4 Mac Mini via Metal:
32.6 tg / 401 pp. Expect the Q3_K variant to be similar or slightly slower
in token generation, similar in prompt processing.
Known issues
Q2_K prompt processing is 7× slower on ROCm
llama.cpp's Q2_K dequant kernel on ROCm (gfx1151) has a known ~7× slowdown
in prompt processing versus other quant types. Metal (Mac) and CUDA (NVIDIA)
are not affected. If you're on an AMD GPU and PP throughput matters, pick the
Q3_K-mixed file instead.
Gemma 4 compression cliff is at ~37 %
We measured earlier ratios of REAP on this model. Above ~30 % pruning, HumanEval
starts degrading noticeably; at 37.5 % pruning it dropped to 75.0 % — a hard
cliff. This is why the release ships at 25 % pruning even though Qwen 3.6
tolerates 48 %.
Reproducibility
This artifact was produced by the Crucible compression toolkit (developed
alongside this model; source not yet publicly released — planned). High-level
pipeline (identical shape to the Qwen 3.6 release):
1# 1. REAP scoring (routing-aware)
2crucible observe google/gemma-4-26B-A4B-it \
3 --samples 512 --seed 42 \
4 -o results/gemma4-base-observation.json
5
6# 2. REAP pruning at 25 %, routing-aware
7crucible compress google/gemma-4-26B-A4B-it \
8 --method reap --routing-aware --ratio 0.25 \
9 --scores-file results/gemma4-base-observation.json \
10 -o outputs/gemma4-reap-25pct
11
12# 3-5. GGUF convert, imatrix, quantize (same steps as Qwen release)
Evaluation methodology
Numbers above are from Crucible's own-harness bench subcommand (no
evalplus / lm-eval runtime dependency). Each problem is run once at
temperature=0; code is executed in a subprocess with a 30 s timeout.
HumanEval numbers use the standard 164 problems (vanilla, not the augmented
humanevalplus). MBPP numbers use the vanilla 500-problem split.
IFEval uses the standard HuggingFaceH4/ifeval dataset with fixed checkers
(paragraph counter that skips markdown dividers, sentence splitter on
punctuation + uppercase, whitespace-normalized repeat_prompt, etc.).
Intended use & limitations
Designed as a coding specialist for local single-user inference on
consumer hardware.
Not designed for or measured on: multilingual generation, image/audio input
(the compressor operates only on the language-model component), long-form
prose, or code in languages other than Python.
The base Gemma 4 model has substantially different training and behavior
from Qwen 3.6 — pick the model that fits your use case:
- Qwen 3.6 REAP-48 Q3K-mixed:
9.4 GB, stronger BFCL tool-calling (91.8 %), also has BigCodeBench
numbers, 3 B active params
- Gemma 4 REAP-25 Q3K-mixed (this repo): 10.3 GB, higher raw HumanEval
(92.7 %), 4 B active params, Gemma-family instruction-following
characteristics
License
Apache-2.0, inherited from
google/gemma-4-26B-A4B-it.
Citation
1@article{lasby2025reap,
2 title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
3 author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and
4 Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
5 journal={arXiv preprint arXiv:2510.13999},
6 year={2025}
7}
