Attention, shared experts, dense MLP projections, lm_head and vision modules are
carried through unquantized.
Bit-exactness
The cast is data-free: NVFP4 scales are derived from the source E8M0 exponents
rather than re-estimated from data. E8M0 has no mantissa, so every source scale is
already an exact power of two. With m = k_max - 8:
E4M3 represents 2^k exactly for k ∈ [-9, 8], so any block within 17 binades of
the tensor max reconstructs bit-for-bit. On Kimi-K3 that covered every block.
No calibration data was used, and none is needed.
Note on precision. Because the per-block scale is always pinned to an
exact power of two, this cast deliberately uses none of NVFP4's precision
advantages at the weight level — neither the finer block-16 granularity nor
E4M3's 3 mantissa bits.
That is the cost of bit-exactness, and it is the right trade: quantization is
irreversible, so a higher-resolution container cannot recover information MXFP4
already discarded. NVFP4's accuracy edge is only reachable by re-quantizing from
BF16, which for 2.8T params means materializing ~5.6 TB. What the conversion buys
is access to the NVFP4 kernel and tooling ecosystem.
Deployment (SGLang)
Verified on the SGLang kimi-k3 branch (not main), 2 nodes × 8×B200, TP16.
Run the identical command on every node, varying only --node-rank.
SGLANG_HOST_IP is mandatory for multi-node — without it the ranks never rendezvous
and startup hangs with no error.
Five non-obvious requirements
Each of these independently prevents the model from serving:
--moe-runner-backend marlin. Kimi-K3 uses the situ (SiTuGlu) activation,
which is not implemented in SGLang's NVFP4 FlashInfer runner
(_SUPPORTED_FP4_ACTIVATIONS = {silu, relu2, gelu}). Marlin supports gated
{silu, situ} and forwards both SiTu parameters
(gemm1_alpha=β=4.0, clamp_limit=linear_β=25.0).
Patch modelopt_quant.py: _SUPPORTED_ACT_STRS += ("situ",). The assert at
:2552 runs before the per-backend dispatch at :2556, so it rejects situ
even when the selected backend supports it. Every backend already applies its own
stricter check afterwards.
Raise UNBALANCED_MODEL_LOADING_TIMEOUT_S (load_model_utils.py, default
480 s, no CLI flag). 1.6 TB over network storage means per-node load skew
routinely exceeds 8 minutes and aborts an otherwise healthy run.
--dist-timeout 3600. Marlin's prepare_moe_nvfp4_layer_for_marlin repacks
247k expert weights, exceeding NCCL's 600 s default; the first rank to finish then
times out alone at SeqNum=1 _ALLGATHER_BASE.
Omit --dcp-size unless FlashInfer is new enough — cutedsl_mla_backend
passes enable_dcp/cp_world to trtllm_batch_decode_with_kv_cache_mla, which
flashinfer 0.6.15 does not accept.
K3 is a reasoning model — it emits thinking tokens before the answer. Use a generous
max_tokens (≥512) and read choices[0].message.content; the thinking is in
reasoning_content. reasoning_effort accepts low / high / max.
bash
1curl$URL/v1/chat/completions -H "Content-Type: application/json" -d '{
2 "model": "k3",
3 "messages": [{"role":"user","content":"What is 17 * 23? Reply with only the number."}],
4 "max_tokens": 1200, "temperature": 0, "reasoning_effort": "low"}'
With too small a budget you get content: "" and finish_reason: "length" — that is
truncated thinking, not a broken model.
This is a W4A16 deployment
Marlin dequantizes the 4-bit weights to BF16 inside the GEMM, so weights stay
4-bit in HBM (the memory and bandwidth win is kept) but the math runs at BF16
tensor-core rate. Activations are never quantized.
Consequences: no activation calibration is needed or used and this deployment setup
does not deliver FP4 compute throughput. Accuracy should be at least that of native
MXFP4, since the weights are bit-identical and the activations are higher precision
than the source's MXFP8.
Evaluation
All three are judge-free and scored with the official harnesses.
IFBench: prompt_loose counts a response only if every constraint on it is met;
prompt_strict was 71.07% and instruction-level loose 78.20%.
SciCode: AA test split, prompt_config=eval/scicode/background, greedy.
Harness: nemo-skills with prompt eval/aai/mcq-4choices and
++eval_type=multichoice, matching the
Artificial Analysis methodology;
198 Diamond questions, greedy for seed 0 and temperature=0.6, top_p=0.95 thereafter.
tokens_to_generate=65536 is load-bearing: at 16384 the model exhausted its budget
inside reasoning on ~10% of questions and emitted an empty answer, scoring as wrong
for a purely harness reason — worth ~8.6 points on the greedy pass (87.4% → 96.0%).
Truncation fell from 9.6% to 0.58%.
config.json reports format: nvfp4-pack-quantized, quant_method: modelopt,
quant_algo: NVFP4, group_size: 16, and an exclude_modules list of 55 globs
derived from the modules that actually kept an unpacked 2-D float .weight.
Branches.main is weight-only (input_activations: null). w4a4 adds
input_scale placeholders of 1.0; they are unused by the marlin path.
Bundled fix
modeling_kimi_linear.py here includes a fix to KimiDeltaAttention: A_log is
stored at head_dim (128) but the upstream release builds it at num_heads (96), so
from_pretrained fails to load its own weights.
The checkpoint holds 96 trained decay values zero-padded to 128 — verified across
all 69 A_log tensors here: elements [96:128] are exactly zero in every one. The
FLA kernel indexes by num_heads, so the padding is inert.
The bundled file follows
discussion #150: it sizes
A_log at num_heads and narrows the checkpoint tensor in _load_from_state_dict,
raising if the discarded tail is ever non-zero.
An earlier version of this file followed our own
discussion #144, which
enlarges the parameter to head_dim instead. That approach loads but cannot run:
fla/ops/kda/gate.py computes A_log.view(H, 1) with H = g.shape[-2] (the head
count), so a 128-element parameter raises
shape '[96, 1]' is invalid for input of size 128 on the first forward. If you pulled
this checkpoint before 2026-07-31, re-fetch modeling_kimi_linear.py.
This matters only for transformers-based loading — SGLang narrows to the first
num_heads elements independently, which the zero-padding makes lossless.
Known limitation: the FP4 fast path
The trtllm-gen cubin pool ships SiTu kernels at NVFP4's 16-element scale block
(57 × BF16-activation, 45 × E2M1-activation), but they are unreachable from SGLang's
NVFP4 runner, which resolves activations via FlashInfer's ActivationType (no SiTu
member) and reads a different cubin store. Wiring the dispatch to
sglang.kernels.ops.moe.trtllm_gen_moe reaches the JIT build stage and then fails on
CUDA 12.8 with namespace "cuda" has no member "maximum" — cuda::maximum requires
CUDA 13's CCCL. Untested beyond that point.
