Variant	Size	BPW	Notes
IQ4_XS (imatrix)	72.0 GB	~4.3	Recommended — importance-matrix-weighted, smaller and typically better than plain Q4_K_M
Q4_K_M	81.5 GB	4.84	Plain k-quant, no imatrix
`imatrix.dat`	217 MB	—	Importance matrix (538 entries) — build your own quants with `llama-quantize --imatrix`

The importance matrix was computed on a German-heavy DE/EN mix matching the pruning calibration (German multi-register human text, German model outputs, English code and math), so the quantization error budget follows the same language priorities as the expert pruning. Both variants are split into ≤45 GB parts; keep the parts of one variant in the same directory and load the first part.

The idea: produce a sparse model in the 100–140B class with a moderate quantization level (Q4_K_M, no aggressive sub-4-bit tricks), so that the quality loss budget is spent once on expert pruning and once on a well-understood quantization — not twice on extreme compression. We plan to spot-benchmark it against dense/sparse models of similar deployed size (e.g. Qwen3.5-122B-class models).

⚠️ Language disclaimer: the expert-pruning calibration was optimized for German and English (German-heavy multi-register calibration mix). Losses in other languages are presumably significantly higher — experts that were mostly used by other languages were preferentially removed. If you need multilingual coverage beyond DE/EN, use the original Inkling-Small.

What was done

Method: REAP (Router-weighted Expert Activation Pruning, Lasby et al., ICLR 2026) — one-shot, no retraining. Per MoE layer, the 128 of 256 routed experts with the lowest router-weighted activation saliency were removed; router rows / e-score bias sliced accordingly. Shared experts, attention, dense layers, embeddings and MTP weights are untouched.
Calibration: ~4M tokens, 56 % German (human-written text across 10 registers — news, legal, parliamentary, scientific, encyclopedic, spoken, easy language, essays, social, narrative — plus German instruction/chat data and German outputs of frontier models), 44 % English code / reasoning / tool-calling / chat.
Quantization: llama.cpp Q4_K_M from BF16.

Calibration only collects routing statistics; no calibration text enters the weights.

Internal retention probes (not benchmarks)

Teacher-forced streaming probes against the BF16 base model (ΔNLL in nats, top-1 agreement with base). Spot checks, not standardized benchmarks:

Probe	ΔNLL vs. base	Top-1 agreement
German prose	+0.39	66.7 %
German factual text	−0.18	77.2 %
English prose	+0.72	73.0 %
Python	+0.68	81.0 %

For comparison: an English-only-calibrated pruning of the same ratio degraded German prose by +1.67 nats. Proper benchmark results (vs. Qwen3.5-122B-class models) are planned.

Usage

Requires a llama.cpp build with Inkling support — currently the add-inkling PR #25731 until it is merged:

bash

git clone -b add-inkling https://github.com/danielhanchen/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m Inkling-Small-REAP-137B-A12B-de-Q4_K_M-00001-of-00002.gguf \
  -p "Erkläre den Unterschied zwischen Bundesrat und Bundestag." -n 300

The split GGUF parts must sit in the same directory; llama.cpp loads them automatically from the first part.

Limitations

DE/EN optimized — other languages likely degrade well beyond typical REAP losses (see disclaimer above).
Code/tool-calling gives up a little quality relative to an English/code-only calibration (that trade-off was chosen deliberately).
Text-only: vision/audio towers of the multimodal original are not part of this GGUF.
No MTP speculative decoding in this GGUF (llama.cpp Inkling MTP support pending). The underlying pruned safetensors checkpoint retains the full MTP stack.
Pruned models remain probabilistic compressions of the original — verify before production use.

About the base model: Inkling-Small

⚠️ Carry-over disclaimer: Everything in this section describes the original thinkingmachines/Inkling-Small (276B-A12B, BF16, multimodal). This repo contains a 50 %-expert-pruned, Q4_K_M-quantized, text-only derivative — the properties and scores below do not all carry over. Treat every number as an upper bound, not a claim about this model. Multimodal inputs are absent here entirely, and multilingual ability outside German/English was deliberately traded away during pruning calibration.

Key facts (original)

Property	Original Inkling-Small
Model type	Multimodal autoregressive transformer (text/image/audio → text)
Architecture	42-layer decoder, sparse MoE FFN: top-6 of 256 routed experts + 2 shared; hybrid local/global attention
Parameters	276B total / 12B active (this repo: 137B total / 12B active)
Context length	up to 1M tokens (long-context behavior of the pruned model is untested)
Numerics	BF16 and NVFP4 (this repo: Q4_K_M GGUF)
Languages	English, with general multilingual capabilities (this repo: DE/EN-calibrated)
Training data	Public, third-party-acquired and synthetic text/image/audio/video (see original card)
MTP	8 speculative multi-token-prediction layers (preserved in our safetensors, not in this GGUF)
License	Apache-2.0

Selected original benchmark scores

Scores of the unpruned BF16 original, from the official model card (where it is compared against Qwen3.5-397B-A17B, MiMo V2.5, Minimax M2.7 and closed models — see there for the full 10-model table). None of these numbers have been re-measured for this pruned+quantized variant.

Benchmark	Original Inkling-Small
SWEBench Verified	80.2 %
Terminal Bench 2.1 (best harness)	64.7 %
SciCode	48.7 %
GPQA Diamond	89.5 %
AIME 2026	95.5 %
HLE (text only / with tools)	31.6 % / 47.8 %
ARC-AGI-1 / ARC-AGI-2	84.0 % / 40.1 %
BrowseComp (with context mgmt)	77.4 %
IFBench	82.2 %
Global-MMLU-Lite	86.7 % (expect larger drops off DE/EN here)
SimpleQA Verified	20.6 %
StrongREJECT (safety)	98.4 %

Multimodal scores (MMMU Pro, Charxiv, MMAU, VoiceBench …) are omitted: the vision/audio towers are not part of this GGUF.

Deployment of the original

The original model has day-0 recipes for SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face transformers (links in the original card), including MTP speculative decoding. This GGUF instead targets llama.cpp (see Usage above); our pruned safetensors checkpoint (TML layout, MTP intact, vLLM/SGLang-compatible) is not published here — open a discussion if you need it.

Safety, bias and limitations (inherited)

The original card documents safety training, known biases, risks and legal terms — all of that applies here at best unchanged and plausibly degraded: expert pruning is a lossy intervention whose effect on safety behavior and refusal calibration has not been separately evaluated. Review the original card's Safety / Bias / Legal sections before production use.

Provenance

Base: thinkingmachines/Inkling-Small (Apache-2.0)
Pruning: REAP streaming pipeline (calibration ran block-wise on a single RTX 5090 workstation without ever loading the 532 GB model into memory)
Per-layer pruned expert lists: see reap_pruning.json in this repo

@inproceedings{lasby2026reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Lazarevich, Sinnadurai, Lie, Ioannou, Thangarasa},
  booktitle={ICLR}, year={2026}
}

Deutsch (Kurzfassung)

REAP-geprunte Variante von Inkling-Small: pro Layer wurden die 128 von 256 Experten mit der geringsten router-gewichteten Aktivierungs-Salienz entfernt (one-shot, kein Training), anschließend Q4_K_M-Quantisierung. Die Kalibrierung war bewusst deutschlastig (56 %, menschliche Texte über zehn Register plus deutsche Instruktionsdaten): deutsche Prosa verliert nur +0,39 nats gegenüber dem Original (statt +1,67 bei englischer Kalibrierung), deutscher Sachtext liegt auf Originalniveau. In anderen Sprachen als Deutsch und Englisch sind die Verluste mutmaßlich deutlich höher. Geplant sind Stichproben-Benchmarks gegen Modelle der Qwen3.5-122B-Klasse.

miweru/Inkling-Small-REAP-137B-A12B-de-GGUF

Inkling-Small-REAP-137B-A12B-de (GGUF)

Variants in this repo