10.40 GB | 28/32 experts per layer | Zero quality loss | GGUF Q4_K_M | Apache 2.0
A losslessly pruned variant of OpenAI's GPT-OSS-20B that removes 4 experts per MoE layer (12.5% of experts) while preserving 100% of benchmark performance. The pruned model saves 1.27 GB (-10.9%) and fits comfortably in 16 GB of RAM for full GPU-resident inference.
Highlights
Lossless pruning: MMLU 78%, GSM8K 92%, Code 80% -- identical to the unpruned original
10.40 GB: fits in 16 GB RAM with room for KV cache and context
~55 tok/s on Apple M4 Pro (Metal GPU)
Drop-in replacement: works with llama.cpp (build 7970+) with no code changes
Reproducible: pruning plan and scripts included
Benchmark Results
Benchmark
Original (11.67 GB)
Pruned (10.40 GB)
Delta
MMLU (0-shot, 100Q)
78%
78%
0 pp
GSM8K (0-shot, 50Q)
70%*
92% (46/50)
--
Code Generation (10Q)
80%
80%
0 pp
Inference speed (M4 Pro)
~55 tok/s
~55 tok/s
negligible
VRAM usage
11.1 GB
~10.0 GB
-1.1 GB
* Original GSM8K was measured on a 10-question subset; the 50-question evaluation was run only on the pruned model. MMLU and Code scores are directly comparable.
1# Download the model2# Place gpt-oss-20b-pruned-90pct.gguf in your models directory34# Run with llama-server (Metal GPU)5llama-server \6 -m gpt-oss-20b-pruned-90pct.gguf \7 --port 8090\8 -ngl 99\9 -c 40961011# Or run with llama-cli for one-shot inference12llama-cli \13 -m gpt-oss-20b-pruned-90pct.gguf \14 -ngl 99\15 -c 4096\16 -p "Explain the theory of relativity in simple terms."
With OpenAI-compatible API
Once llama-server is running, you can query it via any OpenAI-compatible client:
python
1from openai import OpenAI
23client = OpenAI(base_url="http://localhost:8090/v1", api_key="none")45response = client.chat.completions.create(6 model="gpt-oss-20b",7 messages=[{"role":"user","content":"Write a Python function to compute Fibonacci numbers."}],8 max_tokens=500,9 temperature=0.7,10)11print(response.choices[0].message.content)
Apple Silicon (Metal) recommended for best performance; CUDA and CPU also supported
Set max_tokens >= 500 -- the model uses an internal reasoning channel that consumes 100-200 tokens before the final answer
Pruning Methodology
Weight-Based Importance Scoring
Each expert is scored using a static, inference-free heuristic that combines two signals:
Router weight L2 norm: The L2 norm of each expert's column in the router (gate) weight matrix. Experts with larger norms are more likely to be selected by the routing mechanism.
Router bias (softmax-normalized): The sigmoid routing bias term, softmax-normalized across experts within each layer, reflecting the model's learned preference for each expert.
The final importance score for expert e in layer l is:
Experts are ranked by importance within each layer, and the bottom k experts are removed uniformly (4 experts per layer, reducing from 32 to 28). The pruning plan is generated once and applied as a deterministic tensor-level operation on the GGUF file -- no fine-tuning or calibration inference is required.
GGUF Pruning Pipeline
The pruning is performed directly on the GGUF file using reprune_gguf.py:
Parse the pruning plan (JSON) specifying which expert indices to keep per layer
For each MoE layer, copy only the kept expert weight tensors, re-indexing them
Patch the expert_count metadata from 32 to 28
Output a valid, self-contained GGUF file
Pruning Cliff
We found a sharp quality cliff between 28 and 27 experts per layer:
Experts/Layer
Size
MMLU
Notes
32 (original)
11.67 GB
78%
Baseline
28 (this model)
10.40 GB
78%
Lossless
27
10.08 GB
68%
-10 pp sudden drop
25
9.44 GB
52%
Severe degradation
24
9.12 GB
46%
Near-collapse
With only 32 experts per layer (compared to 256+ in DeepSeek-V3 or 512 in Qwen3), each expert carries a disproportionately large share of the model's knowledge. This coarse granularity creates a narrow safe pruning window: 28/32 is lossless, but 27/32 triggers a catastrophic cliff.
Comparison with Original
Original
Pruned (this model)
File size
11.67 GB
10.40 GB (-10.9%)
Experts/layer
32
28
Total experts
768
672 (-12.5%)
MMLU
78%
78%
GSM8K
70% (10Q)
92% (50Q)
Code
80%
80%
Speed
~55 tok/s
~55 tok/s
16 GB RAM
Yes
Yes (more headroom)
Limitations
Narrow pruning window: Due to the coarse 32-expert architecture, only 4 experts can be safely removed per layer. Further pruning causes sharp quality degradation.
Benchmark coverage: Quality was validated on MMLU (100Q), GSM8K (50Q), and code generation (10Q). Performance on specialized domains (e.g., medical, legal, multilingual) has not been evaluated.
No fine-tuning: This is a pruned model with no post-pruning fine-tuning or knowledge distillation. While benchmarks show no degradation, edge-case behaviors may differ from the original.
Reasoning channel token overhead: Like the original, the model uses an internal reasoning channel that consumes 100-200 tokens. Set max_tokens >= 500 for reliable output.
Quantization inherited: Expert weights use MXFP4 (4.25 bits/param) from the original GGUF; no additional quantization was applied.
