This repository contains speculative-decoding drafter checkpoints for Qwen3-8B, including an MTP drafter and a DFly drafter. The drafters propose candidate tokens for the target model to verify, improving inference throughput while preserving the target model's output distribution.
The models were trained with AngelSpec and are intended to be deployed in vLLM together with the Qwen3-8B target model.
For Qwen3-8B, prompts are taken from Open-PerfectBlend (Xu et al., 2024). Responses are regenerated with the Qwen3-8B target model in non-thinking mode, and the resulting target-generated responses are used to train the drafter.
Regenerating responses with the same target model helps the drafter better match the target distribution, which is critical for speculative decoding acceptance rate and end-to-end acceleration.
Training Hyperparameters
These Qwen3-8B draft models use the following training configuration:
Hyperparameter
Value
Batch size
512
Loss schedule
LK loss cold start for 300 steps, then end-to-end TV loss
Learning rate
6.0e-4
Minimum learning rate
6.0e-5
LR decay style
Cosine
Warmup ratio
0.04
Weight decay
0.0
Epochs
10
Maximum sequence length
4096
Evaluation
The table reports acceleration-related metrics across math, code, and chat benchmarks.
Target Model
Drafter
Math500
GSM8K
HumanEval
MBPP
LiveCodeBench
MT-Bench
Avg.
Qwen3-8B
MTP
3.53
3.56
3.33
3.22
3.25
2.57
3.24
Qwen3-8B
DFlash
4.97
5.54
4.77
4.50
4.46
3.16
4.57
Qwen3-8B
DSpark
5.87
6.25
5.56
5.25
5.20
3.77
5.32
Qwen3-8B
DFly
6.06
6.42
5.60
5.34
5.36
3.67
5.41
DFly achieves the highest average result among the listed Qwen3-8B drafters in this evaluation, with an average score of 5.41.
Deployment
Set TARGET to the Qwen3-8B target model and DRAFT to the corresponding drafter repository or local checkpoint path. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.
