kimi-k2.5-eagle3-mla is an Eagle3 MTP draft model with MLA(Multi-Latent-Attention) for accelerating inference of Kimi-K2.5, trained with TorchSpec - an online speculative decoding training framework that runs FSDP training and inference concurrently. If you find this draft model useful, please give our project TorchSpec a star on GitHub.

Why an MLA (Multi-Latent Attention) Draft Model

Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.5 deployment:

Uses less KV cache, which reduces serving memory pressure.
Matches Kimi-K2.5's MLA architecture, so it fits more naturally into the inference engine's KV-cache handling under different serving scenarios such as PD-Disaggregation.

Training Setup

Cluster: 4 nodes x 8x H200 (32 GPUs total)
Training: 2 nodes (16 GPUs), FSDP
Inference: 2 nodes (16 GPUs), Engine (TP=8 per node)
Duration: ~14 hours per phase:

Dataset: Regenerated open-perfectblend dataset

All training responses were regenerated by Kimi-K2.5 via Engine to match the base model's exact token distribution.

Performance

The primary metric is accept_length - the average number of tokens accepted per speculation step with topk=1, num_steps=3, num_draft_tokens=4. Higher is better.

Benchmarks were run using lm_eval.

Category	Benchmark	N	Acc Len
Dialogue	MTBench	80	2.940
Chinese	CEval	212	2.829
Math	GSM8K	500	3.017
Code	HumanEval	164	2.969
Math	MATH500	500	3.051
Math	AIME	30	3.139
VL	MMStar	200	2.597

Quick Start

Requirements

NVIDIA GPU with CUDA 12.0+
vLLM >= 0.18.0, or install the nightly wheel/docker image
The model is supported in SGLang latest main. Refer to the official SGLang installation guide.

Launch Server (vLLM)

bash

vllm serve moonshotai/Kimi-K2.5 \
    --tensor-parallel-size 8 \
    --speculative-config '{"model": "lightseekorg/kimi-k2.5-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
    --trust-remote-code

For deployment configuration, refer to the official vLLM recipes.

Launch Server (SGLang)

bash

sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 8 \
  --trust-remote-code \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla

For deployment configuration, refer to the official SGLang cookbook.

Run Benchmarks

bash

lm_eval \
  --model local-completions \
  --model_args base_url=<url> \
  --tasks gsm8k \
  --batch_size 16