This is an unofficial community derivative of Moonshot AI's Kimi K3 MXFP4 checkpoint. It removes 176 of the 896 routed experts in every MoE layer using router-weighted expert activation pruning (REAP), retaining 720 experts per layer. The checkpoint remains natively multimodal: the vision tower, multimodal projector, attention layers, shared experts, tokenizer, and processor were not pruned or requantized.
Documentation-only update. This revision corrects the model card and evaluation metadata. No model weights or configuration files are changed: all 96 safetensors shards (1,276,563,666,168 bytes total; 1,188.89 GiB) remain as previously published.
Evaluation verdict for the intended workload
Project decision: acceptable for practical use. The full 198-item GPQA-Diamond run produced 155/198 = 78.28% (approximately 78.4% in the project summary). We accept this result for the intended workload of coding, tooling, vision, and agentic use. GPQA-Diamond is retained here as a transparent reasoning datapoint, not as the release gate for this derivative.
This model is not presented as a universal replacement for the strongest frontier reasoning systems. Its purpose is to make Kimi K3's native multimodal, tool-capable behavior practical on a single 8×B200 node after gentle expert pruning, while preserving a 262,144-token serving profile.
GPQA-Diamond context
The comparison below is contextual rather than a leaderboard or a controlled A/B test. These published results use different model versions, prompts, sampling settings, harnesses, and evaluation dates. They are included to show that the REAP-720 result is in the broad neighborhood of several capable frontier or frontier-adjacent systems, not to claim protocol equivalence.
Model / variant
GPQA-Diamond result
Primary source
Kimi-K3-REAP-720-MXFP4
155/198 = 78.28%
This repository's full test split; exact result in evaluation/gpqa_diamond_full.json
The Google model card explicitly warns that provider results are not necessarily directly comparable. The DeepSeek card likewise reports its own decoding and evaluation setup. The exact REAP-720 measurement and raw per-item records remain the authoritative evidence for this checkpoint.
Intended use
The primary target is real-world engineering work, especially:
Coding: repository changes, code generation, debugging, test writing, and long-context software work.
Tooling: structured function calls, terminal-style workflows, API orchestration, and tool-aware task execution.
Vision: native image understanding and vision-in-the-loop coding or engineering workflows; the vision tower and projector are retained unchanged.
Agentic use: multi-step coding and knowledge-work workflows where local deployment, long context, tool access, and controllable latency matter more than maximizing a single academic reasoning score.
Use the model card's serving configuration and run task-specific acceptance tests before production deployment. Formal protocol-matched coding, tool-use, and vision scores are not claimed here; the next validation should use representative real-world workloads in the intended domains.
What changed
Property
Upstream Kimi K3
This checkpoint
Routed experts per MoE layer
896
720
Experts selected per token
16
16
MoE layers evaluated for saliency
92
92
Routed-expert count removed
—
19.64%
Weight format
MXFP4
MXFP4, unchanged for retained tensors
Native maximum position setting
1,048,576
1,048,576
Vision tower / projector
present
not targeted by pruning (vision_tower_modified: false; mm_projector_modified: false)
Checkpoint size
about 1,453 GiB
1,188.89 GiB (96 shards; 1,276,563,666,168 bytes)
REAP saliency was accumulated per expert as sum(router_probability * l2_norm(expert_output)). Calibration streamed one MoE layer at a time as balanced multidomain text plus 16 image-text samples. Each of the 92 scored MoE layers retained its 720 highest-saliency routed experts (a 0.1964 removed fraction). Router rows and expert-correction-bias entries were pruned and renumbered consistently. No retained expert was missing activation coverage in the completed saliency run (retained_experts_never_activated = 0); retained weights remain in mxfp4-pack-quantized format.
This is structural expert pruning, not a newly trained or distilled checkpoint. It should reduce resident weight memory; because K3 still routes to 16 experts per token, it does not by itself reduce the number of active experts used for each token.
Evaluation
Reasoning — verified full-split result
The GPQA-Diamond run used the full 198-item test split and scored 155/198 = 78.28% (approximately 78.4% in the project summary), with 0 infrastructure errors and 0 unscorable outputs; 3 items required a retry. This is accepted for the project's intended coding, tooling, vision, and agentic workload.
This remains a single run (runs = 1), not an average over repeated runs. It is also not a causal paired-pruning estimate because Full K3 was not run through the same prompt/choice/seed/serving harness. The result should therefore be read as an honest full-split REAP-720 measurement and practical acceptance decision, not as a claim of benchmark parity with Full K3.
The published Full-K3 GPQA-Diamond reference is 93.5%, but that number is not a matched control for this run. The project accepts the REAP-720 result because GPQA is not the primary objective; the primary objective is a usable local model for coding, tooling, vision, and agentic workflows.
Machine-readable aggregate and per-item records are included as evaluation/gpqa_diamond_full.json and evaluation/gpqa_diamond_full_records.jsonl.
Vision — invalidated run; no valid final score
There is no valid vision score. A full MMMU-Pro attempt was run but invalidated and excluded from baseline comparison; do not report its 195/246 raw count as a benchmark score.
Invalidated run details (operator-verified, from the correction ledger):
Field
Value
Benchmark
MMMU-Pro
Completed records
246
Correct records
195 (excluded — not a benchmark score)
Retried records
34
Unscorable records
15
Terminal-length finishes
46 of 182 requests in the active server metrics window
Reason invalidated
16,384-token output ceiling caused material length truncation, retries, and unscorable responses
A separate five-item evaluator smoke at 80% exists, but it is diagnostic only — it validates the evaluator and end-to-end generation path, not a floor estimate. It is too small to establish the vision floor. Published MMMU-Pro Full-K3 reference: 81.6%.
Coding — no protocol-matched score
There is no DeepSWE score (final_deepswe_score = null; status: NO PROTOCOL-MATCHED SCORE). An earlier compatibility check found that three generated-code tasks passed the same isolated graders for both REAP-720 and the existing Q2 K3 deployment. A newer same-prompt comparison against K3 via the Moonshot API is summarized below. These are qualitative generation checks, not DeepSWE retention. Published Full-K3 DeepSWE reference: 67.5 (run on the Kimi Code harness, whose exact historical harness revision, task snapshot, seeds, and trajectories are not all public).
Tool use — no terminal score
There is no Toolathlon-Verified score (final_toolathlon_score = null; status: NO TERMINAL SCORE). An official private-mode Toolathlon client (repository hkust-nlp/Toolathlon, commit 2aed2468858f15818acafa178518390cc4b0f5cb) connected its remote execution service to the local OpenAI-compatible endpoint. The selected find-alita-paper task did not reach a terminal score: two long model turns exhausted their output limit, and the job was cancelled after approximately 20 minutes (to free the single-sequence server for GPQA and MMMU-Pro runs). This is compatibility evidence only, not a Toolathlon score or a Full-K3 comparison. Published Full-K3 Toolathlon-Verified reference: 76.5 (full run needs 108 stateful task environments plus external service accounts; the historical K3 trajectories are gated and the immutable configuration behind 76.5 is not public).
Attempt record (evaluation/toolathlon_compatibility.json):
Field
Value
Job ID
job_4419ec6d80d2
Terminal status
cancelled
Model requests started
13
Model responses HTTP 200
12
Completed task artifacts
0
Score
null
Interpretation
protocol compatibility only
Accordingly, no coding/tool retention ratio is reported and no four-dimension pass claim is made.
Benchmark comparability limits
The published Full-K3 references used for the requested floors are 93.5 (GPQA-Diamond), 81.6 (MMMU-Pro), 67.5 (DeepSWE), and 76.5 (Toolathlon-Verified). The coding and tool-use references cannot honestly be compared with the lightweight evidence above: the historical harnesses, task snapshots, seeds, and trajectories behind the DeepSWE and Toolathlon-Verified numbers are not fully public.
Qualitative paired generation spot checks — not formal benchmarks
The same strict prompts were run once through the live REAP-720 deployment and K3 via the Moonshot API. Outputs were saved and tested locally. These single-generation comparisons are useful for examining practical strengths and failure modes, but they are not statistically powered benchmarks, protocol-matched retention measurements, or evidence that the four-dimension quality gate passed.
3/3 compiled; all included and independent tests passed. Working review: 9.8/10
3/3 compiled; all included and independent tests passed. Working review: 9.7/10
Functional tie. REAP supplied broader tests and clearer internal-behavior documentation; the API version was slightly leaner and added stricter zero-capacity/refill validation.
2D aquarium — visual fidelity
7.0/10
7.5/10
API output had the clearer radial crack, water jet/mist, puddle, and mobile scaling. REAP had richer fish, plants, graded water, glass, and settled shards.
2D aquarium — implementation/runtime
8.1/10
7.2/10
REAP had no uncaught runtime errors and stronger pooling. The API output repeatedly produced NaN-geometry console errors.
2D aquarium — equal-weight overall
7.6/10
7.4/10
Narrow REAP win for this sample; both outputs still had Reset/replay correctness issues.
2D aquarium comparison
Side-by-side matched aftermath frames from the K3 API and REAP-720 aquarium generations
Figure: Static matched aftermath frame at approximately 10 seconds. Left animation: generated by K3 via the Moonshot API. Right animation: generated by Kimi-K3-REAP-720-MXFP4 (720/896 routed experts). Both WebGL animations were generated from the same prompt.
What was encouraging about REAP-720 in these checks:
It retained fully runnable Python generation across data structures, concurrency, edge cases, and strict output formatting after 19.64% routed-expert removal.
Its aquarium was the more ambitious desktop illustration, with notably better fish, plants, water treatment, rotating triangular glass fragments, and settled debris.
Its aquarium implementation was cleaner at runtime than the API sample and narrowly won the combined review despite losing the visual-fidelity subscore.
Known REAP aquarium limitations remain: Reset does not fully restore the ambient clock, fish, and bubbles; a live resize does not refresh the renderer dimensions/pixel ratio; the crack can escape the glass; and the mobile composition is weaker. These observations do not change the QUALITY GATE NOT VALIDATED verdict above.
LiveCodeBench (date-windowed direct-code result)
Kimi-K3-REAP-720-MXFP4: 78.85% pass@1 (41/52).
Evaluated on LiveCodeBench release_v6 (code generation), restricted to problems released 2025-03-15 to 2025-04-30 (52 tasks), with greedy decoding (temperature=0, n=1), a 28,000-token generation cap, served with vLLM and scored through the official LiveCodeBench OpenAI-compatible-endpoint harness. This is a single-sample, non-agentic, direct code-generation result (not agentic/tool-use, self-repair, or best-of-n). The run completed all 52 tasks with 0 harness exceptions and 0 timeouts.
Setting
Value
Task
LiveCodeBench code generation
Dataset release
release_v6
Date window
2025-03-15 to 2025-04-30 (inclusive)
Evaluated tasks
52
Metric
pass@1 (greedy)
Decoding
temperature=0, n=1
Max generation length
28,000 tokens
Serving
this checkpoint on vLLM (FP8 KV)
Harness
LiveCodeBench OpenAI-compatible endpoint
Result
41/52 = 78.85%
Context: official LiveCodeBench leaderboard — a DIFFERENT, larger window
These rows are not the same 52 problems. They are the official public leaderboard's default view — 454 problems, 2024-08-01 to 2025-05-01 — recomputed from the leaderboard's own per-problem data and independently verified. They are shown only for rough context and must not be read as a head-to-head ranking against the REAP-720 number above (different task set, sampling, serving stack, and possibly harness).
Model
pass@1
Window / note
Source
O4-Mini (High)
80.2
454 problems, 2024-08 to 2025-05
LiveCodeBench leaderboard
O3 (High)
75.8
same window
LiveCodeBench leaderboard
DeepSeek-R1-0528
73.1
same window
LiveCodeBench leaderboard
Qwen3-235B-A22B
65.9
same window
LiveCodeBench leaderboard
Claude Opus 4 (Thinking)
56.6
same window
LiveCodeBench leaderboard
Claude Sonnet 4 (Thinking)
55.9
same window
LiveCodeBench leaderboard
GPT-4o (2024-08-06)
29.5
same window
LiveCodeBench leaderboard
DeepSeek-V3
27.2
same window
LiveCodeBench leaderboard
Kimi K2 Instruct
53.7
v6, "Aug 24 - May 25" (vendor self-reported)
Moonshot K2 model card
Full Kimi K3 has no published LiveCodeBench score — the official K3 card reports DeepSWE, ProgramBench, Terminal-Bench 2.1, SciCode, etc. instead — so no direct base-vs-pruned LiveCodeBench comparison is available. The REAP-720 result above is a custom recent six-week window and differs from every row in task set, sampling, serving, and possibly harness; treat it as a standalone date-windowed datapoint, not a ranking.
The checkpoint was loaded successfully on one 8× NVIDIA B200 node with the official vllm/vllm-openai:kimi-k3 image. SGLang is not validated for this derivative; only vLLM is claimed.
Observed steady memory was about 168.99 GiB per GPU. A coherent arithmetic probe and multimodal preprocessing path passed. Standard safetensors loading was used; the fastsafetensors loader exceeded available staging memory on this single-node configuration.
A separate concurrent-throughput profile was verified with eight concurrent real MMMU image requests. Relative to the short-context command above, the differing settings were:
Measured result over a 20-second window: 9,420 generation tokens → 471.0 aggregate generation tokens/second, with 8 requests running, 0 waiting, and 0 server errors. This is aggregate concurrent throughput, not single-request decode speed.
Single-sequence mixed workload (verified)
With --max-num-seqs 1, the live vLLM Prometheus counter vllm:inter_token_latency_seconds recorded 124,017 inter-token intervals totaling 1,585.267 s across the 262K validation and GPQA workload: 12.783 ms mean inter-token latency, or 78.23 decode tokens/s. This is a Prometheus aggregate over mixed successful requests (including the 260K-context validation and the official-protocol GPQA evaluation), not a synthetic fixed-length throughput suite. The machine-readable counter snapshot is evaluation/throughput_262k.json.
The same checkpoint passed a 262,144-token FP8-KV boot and a near-limit end-to-end request. Relative to the short-context command, the verified long-context settings are:
The request contained 260,095 API-counted prompt tokens (260,094 locally counted) plus 53 completion tokens and finished normally in 54.78 s with finish_reason="stop", returning READY. Machine-readable evidence: evaluation/long_context_262k.json.
For this B200 image, forcing the FLASHINFER MLA prefill backend is required for FP8 prefill-query quantization; the auto-selected FlashAttention MLA prefill backend does not support that path.
Revision caveat. The derivative config contains no embedded commit hash or source revision. Do not claim the build source is pinned to the observed public parent revision above; source_revision_verified is false. That revision is recorded only as the public parent state observed on 2026-08-01.
The pruning procedure is recorded in REAP_PRUNING.json (schema version 1, created_utc 2026-08-01), including SHA-256 hashes of the scripts and reference baselines for reproducibility:
The published reap_tools/eval_reap.py bytes hash to the value shown above, and that value matches REAP_PRUNING.json. A later unpublished local experiment increased the vision output budget to 32,768 tokens, so it has different evaluator bytes; no result from that modified evaluator is represented as a completed benchmark in this repository.
Recovered expert keep map
The exact retained-expert map is published as:
provenance/kimi_k3_reap720_keep_map.json — human-readable artifact with per-layer new→old and old→new mappings, layer hashes, shard names, and validation statistics.
Recovery method: for each of the 92 MoE layers, every retained BF16 router row in gate.weight was matched byte-exactly against the 896 parent rows at the observed public parent revision. All 66,240 retained rows had exactly one parent match, were strictly ascending, and were injective. The corresponding mapped F32 e_score_correction_bias entries were then verified byte-for-byte for all 66,240 mappings. This establishes an empirical match to public parent revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569; it does not establish build-time source attestation, so source_revision_verified remains false.
This checkpoint is intended primarily for coding, tooling, native vision, and agentic workflows where local deployment, long context, and practical tool interaction matter. It preserves Kimi K3's multimodal architecture while reducing resident weight storage enough for a single 8×B200 deployment. The project's full-split GPQA-Diamond result of 155/198 = 78.28% (approximately 78.4% in the project summary) is accepted as a practical reasoning datapoint for these intended workloads.
Expert removal can change routing behavior and may degrade particular languages, domains, visual tasks, long-context behavior, or agent trajectories even when short tests look normal. Run task-specific acceptance tests before production use, especially for the coding repositories, tools, images, and agent loops that matter to your deployment.
This repository does not claim safety improvements, alignment changes, or a new training procedure. The upstream Kimi K3 model limitations continue to apply.
License and attribution
The checkpoint and copied upstream code are provided under the Kimi K3 License. This derivative retains Moonshot AI's copyright and license notice. Users are responsible for reviewing the license, including its conditions for certain large commercial products and Model-as-a-Service businesses.
