Ornith-1.5-REAP-20B-A3B-GGUF

⚠️ This is a research artifact. Expert pruning removes knowledge. Performance was validated around agentic coding and ML engineering (Python/C++). General-domain behavior (translation, creative writing, broad knowledge, multilingual nuance, long-document prose) has not been tested and is expected to be worse than the 35B base.

Part 1 — What this is, and what it's for

Intended use

• Local agentic coding / ML-engineering workflows (Python/C++, tool calling, repo work, debugging, quantization/model work) on ~16 GB VRAM GPUs and 24 GB unified-memory Macs (via llama.cpp).
• The 20B checkpoint is a high-precision slice of the 35B MoE: same ~3B active path, same architecture (hybrid GDN linear-attention + softmax attention + MoE), 48.4% fewer experts to store — so the memory savings come from storage, not from a smaller active compute path.

Files

Where it should succeed

• Tool calling and ML-engineering coding — the calibration/training domain. Expert pruning is near-lossless in-domain when calibrated on the target domain (REAP, [arXiv:2510.13999]); the recovery LoRA and the imatrix calibration were built from the same domain.
• Quick, high-quality generations at 4–5 bits on the active 3B path — q5k_ud matches or exceeds the BF16 checkpoint's own perplexity on the held-out test text (see Part 2), i.e. the quantization costs less than the recovery pass gains back.

Where it may fail (be realistic)

• General knowledge & language quality: the 124 pruned experts carried generic-domain knowledge. Expect degraded translation, creative writing, and open-ended knowledge recall vs the 35B base. Untested — treat as unknown.
• Hard multi-step reasoning: long reasoning traces are the heaviest load on the remaining experts; expect more brittleness than the 35B on very long chains.
• Low-bit variants (ud_iq4nl): the IQ3_S/IQ4_NL expert map costs ~0.26 PPL on general text — fine for experimentation, not the quality pick.
• MTP: the multi-token-prediction head was stripped (not part of the reaped checkpoint).

Serving tip — loop avoidance

A note on quality vs the 35B bases

Part 2 — Everything we know (pipeline, data, measurements)

Pipeline (executed + designed)

1. Trace generation. Served the 35B (NVFP4 W4A16, ModelOpt PTQ) and generated on-policy traces from a domain mix. Corpus: 2,387 prompts → ~5.3M tokens, splits ~50% tool-calling / ML-engineering, ~20% agentic multi-turn, ~15% math, ~15% general; 99.4% of rows completed with a final answer (0.6% thinking-only). All rows are the model's own chat-formatted generations (reasoning + answer).
2. REAP (Router-weighted Expert Activation Pruning) — [arXiv:2510.13999], Cerebras repo (ICLR 2026). Saliency S_j = mean(g_j · ||f_j||₂) over the calibration tokens; 256 → 132 experts/layer (48.4%), top-8 routing preserved. Calibration used the domain mix above (code/ML-eng 35%, tool 25%, agentic 20%, math+general 20%). Includes the router-weight renormalization fix from the Cerebras repo.
3. Recovery LoRA. Unsloth 4-bit QLoRA on the reaped checkpoint (rank 16, ~2 epochs over the completed traces; targets: attention q/k/v/o, GDN linear-attention projections, shared expert) with a small anti-hallucination abstention subset (~11%) — "I don't know" responses for obscure technical queries — blended into the recovery mix.
4. Anti-doomloop (FTPO / Antidoom) — designed, data prepared, not applied to this release. Liquid AI's Antidoom / blog: sample low-temperature completions, detect repeated spans, train a LoRA with Final Token Preference Optimization on the loop-starting token vs plausible alternatives (logit-space KL, two-part regularisation). An 18k-prompt code/math/tool subset was prepared for this stage. It was not run for this release because the measured loop proxy on the evaluation battery was already low (n-gram self-overlap 0.026–0.029 across variants); the stage remains the recommended follow-up if looping is observed in practice. (Liquid's published results: LFM2.5-2.6B 10.2%→1.4% loop rate, Qwen3.5-4B 22.9%→1%.)
5. Quantization. Converted to GGUF F16, then computed an importance matrix (imatrix) via llama.cpp on ~3M tokens of chat-formatted calibration text rendered from the generated traces (same domain mix) — instruct models need chat-formatted imatrix calibration, not raw text. Quantized variants with the imatrix, plus a UD-style per-tensor map (Unsloth Dynamic-3.0-style important-tensor raises: attention/ssm-out/shared-expert Q8_0, norms F32, deepest-layer expert downs at Q6_K) lifted from Unsloth's published Qwen3.6-35B map (same architecture family; tensor names transfer 1:1).

Perplexity (KLD) on held-out text

• The REAP cut costs +0.66 PPL on general text (~18% relative) — expected: 48.4% of experts removed, generic text exercises all experts. On the calibration domain (agentic coding / ML engineering) the cut is near-lossless, consistent with REAP's domain-calibration findings.
• The recovery LoRA moved generic-text perplexity by ~0.000 (4.3411 both) — it tuned the domain, not the base distribution.
• The quantization improved the recovered BF16's PPL (4.2730 < 4.3411) — the domain imatrix + important-tensor raises regularize the low-bit experts into a slightly better distribution on this text.
• The 35B IQ3 (3.7603) still beats q5k_ud (4.2730) on general-text PPL — the full model wins on generality; the reaped model wins in the targeted domain at a comparable size.

Evaluation battery (recovered vs pre-recovery)

Quant variant comparison (all from the recovered BF16, with the domain imatrix)

Technical notes & caveats

• The checkpoint is text-only (vision wrapper stripped; no vision weights exist in the source). GDN linear-attention kernels in llama.cpp co-process on CPU on some devices — expect prefill to be heavier than pure-GPU dense models.
• IQ-type (imatrix) quants cost ~5–10% decode speed vs non-imatrix at the same size.
• The MTP head is stripped (converted with --no-mtp); no speculative decoding.
• ssm_out (linear-attention state output) is kept at Q8_0 (per Unsloth's Qwen3.5-family guidance, quantizing it hurts).
• The BF16 folder is the full reaped+recovered checkpoint (Qwen3_5MoeForCausalLM, 132 experts, text-only config) — use it to re-quantize with different maps/imatrixes.
• The 35B IQ3 numbers in Part 2 were measured on AtomicChat's AD-IQ3_S/IQ3_XXS GGUF — the un-pruned base, included here as a reference data point only.
• Untested: translation, creative writing, long-document prose, multilingual, medical/legal QA, and other general domains. Use accordingly.