Hy3-REAP-200B-21B-GGUF
A 33%-expert-pruned version of Tencent Hunyuan Hy3 (295B / A21B), compressed with REAP (Router-weighted Expert Activation Pruning) — pruning only, no expert merging.
This checkpoint reduces the number of routed experts in
tencent/Hy3 from
192 → 128 per MoE layer (−33%) using the
REAP saliency criterion, as implemented in the
SamsungSAILMontreal/ream codebase. The result is a smaller, drop-in Hunyuan V3 model (~200B total parameters) that keeps the
same 21B active-parameter path and the full 256K context window.
Because the router is left untouched and no experts are averaged together, this is a pure REAP checkpoint — the pruned model retains the original router's independent, input-dependent control over the surviving experts.
Model Overview
| Property | Value |
|---|
| Base model | tencent/Hy3 (Hunyuan V3, 295B / A21B) |
| Architecture | HYV3ForCausalLM (model_type: hy_v3) |
| Compression method | REAP (saliency = reap, merging = none) |
| Pruning ratio | ~33% of routed experts (192 → 128 per layer) |
| Total parameters | ~200B (down from ~295B) |
| Active parameters | ~21B (unchanged) |
| Routed experts / layer | 128 (was 192), top-8 routing |
| Shared experts / layer | 1 (unchanged) |
| Layers | 80 (1 dense + 79 sparse) |
| Context length | 256K (max_position_embeddings = 262144) |
| MTP layers | 1 (num_nextn_predict_layers = 1) |
| Precision | BF16 |
| Calibration data | OpenMOSE/reap-calib-mix |
The active-parameter path is unchanged because top-8 routing and the single shared expert are preserved; pruning removes only rarely / weakly used routed experts, so per-token FLOPs stay the same while the total memory footprint drops by roughly one third.
Method: REAP (pruning, not merging)
REAP (Router-weighted Expert Activation Pruning) scores every expert on a calibration set using a saliency signal that combines
router gate-values (how strongly / often the router selects an expert) and
expert activation norms (the magnitude of each expert's output contribution), normalized by the number of tokens routed to that expert. Experts with the lowest saliency contribute least to each layer's output and are removed.
REAP is a one-shot, post-training procedure: no gradient updates or fine-tuning are required after pruning. Crucially, unlike expert-merging approaches, pruning does not collapse experts into shared weights, so the router keeps its fine-grained, input-dependent modulation of the remaining experts. The REAP authors show this matters most on generative tasks, where merging introduces an irreducible reconstruction error.
Why REAP rather than REAM on Hy3
The
ream codebase supports both REAP (pruning) and
REAM (Router-weighted Expert Activation Merging). REAM often preserves multiple-choice accuracy better by
merging grouped experts instead of dropping them.
For Hy3 specifically, we found that REAP gave the better trade-off between quality loss and achievable pruning depth than REAM. At the compression ratio used here, the pruning-only path stayed closer to the baseline on generative behavior while allowing a cleaner one-third reduction, so this release ships as a pure REAP checkpoint (merging: none).
Calibration data
Off-the-shelf calibration corpora left the expert-saliency estimates unbalanced across domains. To improve pruning accuracy, calibration used
OpenMOSE/reap-calib-mix — the same mixed dataset used in the OpenMOSE model-distillation pipeline — which balances the domain distribution seen during saliency accumulation and produced a more even expert-importance estimate.
Pruning configuration
The run used the following key arguments (from config.json → merge_args):
| Argument | Value |
|---|
saliency | reap |
merging | none (pure pruning) |
grouping | ream |
group_size | 16 |
merge_size | 128 (target experts per layer) |
dataset | reapmix_pack (OpenMOSE/reap-calib-mix) |
calib_size | 3072 |
calib_seq_len | 2048 |
batch_size | 8 |
mix_ratio | 1.0 |
no_gated_sim | true |
no_sequential | true |
Architecture: before vs. after
| Field | Base Hy3 | Hy3-REAP-200B-21B |
|---|
num_experts | 192 | 128 |
num_experts_per_tok | 8 | 8 |
num_shared_experts | 1 | 1 |
num_hidden_layers | 80 | 80 |
hidden_size | 4096 | 4096 |
moe_intermediate_size | 1536 | 1536 |
expert_hidden_dim | 1536 | 1536 |
intermediate_size (dense) | 13312 | 13312 |
first_k_dense_replace | 1 | 1 |
num_attention_heads | 64 | 64 |
num_key_value_heads | 8 | 8 |
head_dim | 128 | 128 |
qk_norm | true | true |
moe_router_use_sigmoid | true | true |
router_scaling_factor | 2.826 | 2.826 |
rope_theta | 11158840.0 | 11158840.0 |
max_position_embeddings | 262144 | 262144 |
vocab_size | 120832 | 120832 |
tie_word_embeddings | false | false |
Everything except the routed-expert count is preserved, including the router, attention, shared expert, dense layer, and RoPE configuration.
Evaluation
| Benchmark | Base Hy3 | Hy3-REAP (33%) | Δ |
|---|
| MMLU | 85.65% | 82.37% | −3.28 pt (−3.8% rel.) |
| GSM8K | 94.54% | 94.62% | +0.08 pt (≈ noise) |
At 33% expert pruning, MMLU drops by only a few points while GSM8K is effectively unchanged (within run-to-run noise). Grade-school math reasoning survives the compression almost perfectly, and broad knowledge (MMLU) degrades modestly.
A note on Hy3's pruning sensitivity
In our experiments, Hy3 is considerably less pruning-tolerant than other open MoE models such as Qwen3 and GLM. Those families can typically absorb deeper expert pruning (often up to ~50%) with small quality loss, whereas Hy3 leaves very little headroom: quality degrades sharply beyond the ratio used here. In practice, ~33% is close to the useful pruning ceiling for Hy3 — this checkpoint sits near that ceiling deliberately, and we do not recommend pushing the expert count much lower without recovery training. The careful calibration mix (reap-calib-mix) was necessary precisely because of this low tolerance.
Benchmark figures are self-reported and intended for relative comparison against the base model under an identical evaluation harness; absolute numbers may differ from other setups.
Usage
The model uses the standard Hunyuan V3 (hy_v3) architecture and is a drop-in replacement for tencent/Hy3 at reduced size. Serve it the same way you would serve Hy3.
Transformers
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_id = "OpenMOSE/Hy3-REAP-200B-21B"
5tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 model_id,
8 torch_dtype=torch.bfloat16,
9 device_map="auto",
10 trust_remote_code=True,
11)
12
13messages = [{"role": "user", "content": "Explain mixture-of-experts pruning in one paragraph."}]
14inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
15out = model.generate(inputs, max_new_tokens=512)
16print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
Hy3's hybrid fast/slow reasoning modes (no_think / think_low / think_high) and tool-calling behavior are inherited from the base model.
Limitations
- One-shot compression, no recovery training. This checkpoint is pruned only; no post-pruning distillation or fine-tuning was applied. Optional expert-wise distillation could recover some of the MMLU gap if needed.
- Near the pruning ceiling. Hy3 tolerates pruning poorly (see above). Do not expect deeper pruning of this architecture to remain lossless.
- Benchmark coverage is narrow. Only MMLU and GSM8K were measured here. Long-context retrieval, coding, agentic tool use, and multilingual behavior were not re-evaluated after pruning and may be affected differently.
- Inherited behavior. All safety, bias, and factuality characteristics of the base Hy3 model carry over and may interact with pruning in ways not fully characterized.
License
This is a derivative of
tencent/Hy3, which is released under the
Apache License 2.0 (the full Hy3 release removed the regional restrictions present in the earlier
Hy3-preview). This checkpoint is distributed under the same terms. Users should confirm they are complying with the base model's license before deployment.
Acknowledgements & Citation
Pruning was performed with the
SamsungSAILMontreal/ream codebase, which implements both REAP and REAM. The base model is Tencent Hunyuan Hy3.
If you use this checkpoint, please cite the REAP paper (pruning criterion), the REAM paper (codebase / merging counterpart), and the base Hunyuan Hy3 model.
1@article{reap2025,
2 title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
3 author = {Cerebras Research},
4 journal = {arXiv preprint arXiv:2510.13999},
5 year = {2025}
6}
7
8@article{ream2026,
9 title = {REAM: Merging Improves Pruning of Experts in LLMs},
10 author = {Samsung SAIL Montreal},
11 journal = {arXiv preprint arXiv:2604.04356},
12 year = {2026}
13}
Pruned and released by OpenMOSE. 
