recipe/ additionally contains everything needed to upgrade this checkpoint locally into the FP8-attention + FP8-shared-expert build we benchmarked (that build is ~1.6 TB and does not fit this account's storage quota, so it ships as a reproducible recipe rather than weights).

⚠️ Required engine fix (vLLM ≤ 0.28.0)

Every vLLM release through 0.28.0 mis-composes the SITU activation scale for K3 NVFP4 and emits garbage on the default fp4 path. See recipe/vllm-0.28.0-situ-scale-fix.patch. The subtlety: the loader composes that scale in three places, and the third — a post-load recompute — silently overwrites the earlier fixes, which is why partial patches look like they "don't take". Upstream PR #53132 (main since 2026-08-21) ships in v0.28.1.

Until you run 0.28.1+: apply the patch, or use --kernel-config '{"moe_backend":"marlin"}' and accept dequant-class speed.

Serving (8×B300 / SM103, vLLM 0.28 + fix)

bash

vllm serve Weyaxi/Kimi-K3-NVFP4-W4A4 \
  --served-model-name kimi-k3-w4a4 --tokenizer moonshotai/Kimi-K3 \
  --trust-remote-code --limit-mm-per-prompt '{"image":0,"video":0}' \
  --tensor-parallel-size 8 --gpu-memory-utilization 0.92 \
  --max-model-len 1048576 --kv-cache-dtype fp8 \
  --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' \
  --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 \
  --kernel-config '{"moe_backend":"flashinfer_trtllm"}' --enable-expert-parallel \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
  --max-num-batched-tokens 8192 --max-num-seqs 512

The recipe: quantizing attention and the shared experts

To our knowledge no other K3 checkpoint quantizes these. recipe/scripts/ does it in place:

step	script	what it does
1	`attn_fp8_inplace.py`	FP8 W8A8-dynamic channel quant of 579 attention tensors, `(out,1)` scales
2	`se_fp8_inplace.py`	same for 279 shared-expert + dense-layer-0 tensors
3	`build_rung2_config.py`, `build_se_config.py`	rebuild `quantization_config` groups + `ignore` list
—	`config.fp8attn-fp8se.json`	the resulting config, for reference
—	`fetch_`, `splice_`	HTTP range-fetch / atomic in-place shard rewrite helpers

Hard constraint we hit: kv_b_proj and o_proj (117 tensors) receive already-FP8 activations (FP8 KV cache + prefill query quantization). No generic kernel accepts FP8 inputs there, so they must stay BF16 — they are in the ignore list.

Quality after both steps, measured against the BF16-activation baseline: GSM8K-platinum 99.00 (n=200, 0 errors), tool-calling 60/60, needle 36/36 across 32k/80k/120k.

Honest performance

Kimi-K3 native is already 4-bit (MXFP4 weights), so this requant is not a bandwidth win. We benchmarked it same-session, both sides, with native tuned as hard as the quant — our sweep found native's own vLLM-0.28 optimum (kernel autotune ON, 8k prefill chunks) is worth +8–78% over the day-0 recipe, and we adopted those wins for native before comparing.

Against that tuned baseline, on one 8×B300 node:

workload	quant vs native
Per-stream pacing (TPOT) at heavy saturation, c=128	1.05–1.13× (ours)
Real text, ShareGPT c=128	0.95–1.01× agg, TPOT 1.08× ours
Standard aggregate throughput	0.89–0.97× (native)
Batched decode c=64–128, n=8 sampling	0.70–0.85× (native)
Cache-heavy agentic / RAG	0.87× / 0.97×, cached TTFT ~1.5× (native)
Rate-paced 3–7 req/s	native at every rate
KV pool	−8.7% (native)

Run native for throughput-billed work, batch/RL farms, cache-heavy serving and single-stream. The quant earns its place at per-stream latency under saturation — and as evidence that a full W4A4 + FP8-attention + FP8-shared-expert requant of a 2.8T MoE preserves quality exactly, which matters most for model families whose native weights are not already 4-bit.

Earlier versions of this card claimed larger wins (1.4–1.5× batched decode). Those were measured against native's day-0 config; they did not survive tuning native properly. These numbers did.