Model Architecture: LlamaForCausalLMEagle3 (Eagle3 MTP draft model)
- Input: Text
- Output: Text
Supported Hardware Microarchitecture: AMD Instinct MI355X
ROCm: 7.0.0
PyTorch: 2.9.0
pytorch-triton-rocm: 3.5.0
Transformers: 4.57.0
Operating System(s): Linux
Inference Engine: vLLM
Model Optimizer: AMD-Quark) (v0.12)
- Quantized layers: midlayer.self_attn.{q,k,v,o}_proj, midlayer.mlp.{gate,up,down}_proj
- Weight quantization: FP8 E4M3, static, per-channel, symmetric, channel axis 0
- Activation quantization: FP8 E4M3, dynamic, per-channel, symmetric, channel axis 1
Calibration Dataset: None (file-to-file quantization; no calibration required)

Kimi-K2.5-Eagle3-FP8 is an FP8-quantized version of lightseekorg/kimi-k2.5-eagle3, an Eagle3 MTP draft model for accelerating inference of moonshotai/Kimi-K2.5 with speculative decoding. This checkpoint was quantized with AMD Quark. The quantized tensors use FP8 quantization metadata in the model config, and the LM head is not quantized (intentionally excluded from quantization).

Model Quantization

The model was quantized from lightseekorg/kimi-k2.5-eagle3 using AMD Quark. The checkpoint keeps the original Eagle3 architecture and exports Quark quantization metadata in config.json. The fc projection and lm_head are intentionally not quantized.

Quantization scripts:

bash

cd Quark/examples/torch/language_modeling/llm_ptq/

python3 quantize_quark.py \
  --model_dir lightseekorg/kimi-k2.5-eagle3 \
  --quant_scheme ptpc_fp8 \
  --exclude_layers fc lm_head \
  --output_dir amd/Kimi-K2.5-Eagle3-FP8 \
  --file2file_quantization

For further details or issues, please refer to the AMD Quark documentation or contact the respective developers.

vLLM Loading Note

When using this FP8 Eagle3 checkpoint as a vLLM draft model, make sure the exported config.json records the excluded layers as regex patterns. If Quark exports:

json

"exclude": [
  "fc",
  "lm_head"
]

change it to:

json

"exclude": [
  "re:.*fc.*",
  "re:.*lm_head.*"
]

Intended Use

This model is intended to be used as an Eagle3 draft model for speculative decoding with moonshotai/Kimi-K2.5 as the target model.

Because this is an AMD Quark FP8 checkpoint, make sure your inference runtime supports the quantization format and Eagle3 speculative decoding before deployment. Please validate quality and acceptance length in your own serving stack.

Throughput

Kimi K2.5 Eagle3: BF16 and AMD Quark FP8 Drafts — amd/Kimi-K2.5-MXFP4 target, ISL/OSL = 1K/1K, TP=4 on a single AMD Instinct MI355X node.

Concurrency	No-spec (tok/s/GPU)	BF16 Eagle3 (tok/s/GPU)	FP8 Eagle3 (tok/s/GPU)
4	82.7	157.0 (1.90x)	165.2 (2.00x)
8	142.2	269.1 (1.89x)	270.1 (1.90x)
16	220.5	399.6 (1.81x)	412.7 (1.87x)
32	342.2	627.6 (1.83x)	633.8 (1.85x)
64	533.3	901.6 (1.69x)	936.6 (1.76x)

Across all tested concurrencies, the AMD Quark FP8 Eagle3 draft matches or exceeds the BF16 draft throughput, reaching up to 2.00x over the no-spec baseline.

Reproduction

The throughput numbers above were produced with vLLM on a single AMD Instinct MI355X node. Three models are involved:

Target model: amd/Kimi-K2.5-MXFP4
BF16 draft model: lightseekorg/kimi-k2.5-eagle3
FP8 draft model: amd/Kimi-K2.5-Eagle3-FP8 (this model), quantized with AMD Quark FP8 metadata and sharing the BF16 target LM head.

In this setup, the FP8 draft path dispatches through vLLM RowWiseTorchFP8ScaledMMLinearKernel — i.e. torch._scaled_mm over hipBLASLt row-wise scaled FP8 GEMM — rather than the AITER preshuffled FP8 path. The target MXFP4 model uses the ROCm FP4 ASM path via VLLM_ROCM_USE_AITER_FP4_ASM_GEMM=1.

Docker Image

vllm/vllm-openai-rocm:nightly-fb1ac806c55a6dc96fe92261b80c8550e9c39d2f

Serving (FP8 Eagle3 draft)

Launch the FP8 container (standard ROCm device mounts), then start the vLLM server. This example matches the TP=4, ISL/OSL = 1K/1K sweep:

bash

docker run -it --rm \
  --device /dev/kfd --device /dev/dri --group-add video \
  --ipc host --shm-size 16g --network host \
  vllm/vllm-openai-rocm:nightly-fb1ac806c55a6dc96fe92261b80c8550e9c39d2f \
  bash

# Inside the container:
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FP4_ASM_GEMM=1       # target MXFP4 FP4 ASM GEMM
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_RMSNORM=0            # required for TP < 8

vllm serve amd/Kimi-K2.5-MXFP4 \
  --port 8888 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 2304 \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --speculative-config '{"model":"amd/Kimi-K2.5-Eagle3-FP8","method":"eagle3","num_speculative_tokens":6,"draft_tensor_parallel_size":1}'

BF16 draft baseline: use image vllm/vllm-openai-rocm:v0.19.0, --max-model-len 2248, and "model":"lightseekorg/kimi-k2.5-eagle3" in the speculative config.
No-spec baseline: drop --speculative-config entirely.

Benchmarking

With the server up, run a vllm bench serve throughput sweep over concurrency C ∈ {4, 8, 16, 32, 64} (10 prompts per concurrency; input/output lengths are sampled around the 1K target with --random-range-ratio 0.8, and --ignore-eos forces each request to emit its full sampled output length):

bash

for C in 4 8 16 32 64; do
  vllm bench serve \
    --model amd/Kimi-K2.5-MXFP4 \
    --backend vllm \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 1024 \
    --random-range-ratio 0.8 \
    --num-prompts $((C * 10)) \
    --max-concurrency "$C" \
    --request-rate inf \
    --ignore-eos \
    --use-chat-template \
    --trust-remote-code
done

Throughput is reported as decode tokens/s per GPU (total throughput divided by the 4 GPUs); speedups in parentheses are relative to the no-spec baseline at the same concurrency.

Citation and Acknowledgements

This model is derived from lightseekorg/kimi-k2.5-eagle3. Please refer to the source model card for the original training details, benchmarks, and acknowledgements.

amd/Kimi-K2.5-Eagle3-FP8

Model Overview

Model Quantization

vLLM Loading Note

Intended Use

Throughput

Reproduction

Docker Image

Serving (FP8 Eagle3 draft)

Benchmarking

Citation and Acknowledgements

License

amd/
Kimi-K2.5-Eagle3-FP8