Inferact/Kimi-K3-DSpark is an MLA-native DSpark draft model that accelerates Kimi-K3 on vLLM, served through vLLM's native dspark speculative method.
The draft trains on target hidden states extracted from vLLM itself — the same engine that serves it, so the numerics it learns from are the numerics it meets at inference. TorchSpec provides the loop, streaming those hidden states from live target inference into concurrent FSDP draft training.
DSpark = a block-diffusion backbone of 5 dense layers with non-causal attention, drafting 7 tokens in a single parallel pass, a low-rank sequential Markov head supplying the intra-block dependency, and a confidence head for resource-aware scheduling. Mirroring Kimi-K3's own MLA attention means draft and target share one KV layout (a compact 576-element latent per token), so the draft's pages unify with the target's cache — KV offloading and P/D disaggregation work with no separate page format.
Performance
Peak bs=1 decode: 464 tok/s
Under low-entropy real reasoning workload — Kimi-K3 + DSpark on vLLM can achieve 464 tok/s using the public vllm/vllm-openai:kimi-k3 image on 4 × GB300 at bs=1 & tensor-parallel-size=16.
Speculator acceptance
Speculators do best on predictable, low-entropy work like the reasoning workload behind the 464 tok/s above, and worst on open-ended, high-entropy generation. So we measured 14 benchmarks spanning math, code synthesis, real-world software engineering, multi-turn chat, RAG and QA, multilingual text, creative writing and long-context generation — all with the Kimi-K3 chat template enabled and production sampling parameters, with temperature=0 reported alongside for reproducibility. Acceptance stays strong at long context too, verified on AA-LCR's ~95k-token multi-document prompts. That is the most comprehensive view we can give of how the speculator performs on real-world workloads.
Data — all responses regenerated on-policy by Kimi-K3 itself, so the draft learns the target's own reasoning traces and chat formatting. Prompts come from public datasets:
A nine-category prompt pool from the NVIDIA Nemotron SFT/RL collections and CohereLabs/aya_dataset, spanning chat, code, multilingual, RAG/QA, math, structured output and safety. Evaluation-set prompts are excluded.
Method: DSpark with block_size=7, trained on a combined CE + L1 distribution-distillation objective against the target's post-final-norm hidden state, bf16. The draft consumes Kimi-K3 auxiliary hidden states from target layers (2, 23, 47, 71, 89) of 93. Roughly two epochs in total, on GB300 nodes.
Draft architecture, block size, sequence length, and loss weights are YAML-configurable — see the TorchSpec repo.
Quick Start
Requirements
For serving Kimi-K3 itself — hardware, parallelism and engine flags — follow the official vLLM recipe: recipes.vllm.ai/moonshotai/Kimi-K3.
vLLM offers two knobs on top of the defaults, both used in the temperature=1.0 column above:
draft_sample_method — probabilistic samples the draft from its own distribution instead of taking its argmax. Pair it with a sampling client; use greedy when serving at temperature=0 so the draft matches the client.
rejection_sample_method — block verifies the drafted block as a unit rather than token by token. It is a no-op under greedy decoding (at temperature=0 verification reduces to a deterministic argmax match), so it only applies to the sampling configuration.
The temperature=0 column was produced with {"draft_sample_method": "greedy"} and no rejection_sample_method.
