EAGLE3 draft model for Qwen3.5-9B, trained on a decontaminated instruction-mixture corpus. Use it with SGLang (or other EAGLE3-compatible runtimes) to accelerate autoregressive decoding via speculative sampling.

Model Overview

Item	Value
Architecture	`LlamaForCausalLMEagle3` (1-layer draft head)
Target model	Qwen3.5-9B
Draft vocab size	50,000
Aux hidden states	Layers 1, 15, 28
Precision	bfloat16

Training Data

Training uses openeurollm/open-perfectblend-decontaminated, a decontaminated variant of mlabonne/open-perfectblend. Benchmark overlap (e.g. HumanEval, MATH, GPQA, LiveCodeBench) was removed before training to reduce evaluation contamination.

The dataset was converted to a single ShareGPT-style parquet file (role / content turns) for training.

Training Framework

Trained with mini-specforge (link), a lightweight EAGLE3 training pipeline built on PyTorch FSDP and Hugging Face Transformers.

Hyperparameter	Value
Learning rate	1e-4
Batch size	4
Max sequence length	2048
Epochs	1
TTT length	7
Warmup ratio	0.015
Draft vocab mapping	Top-50k tokens from UltraChat (Qwen3.5 tokenizer)

SGLang Evaluation

Benchmarked with SGLang EAGLE3 speculative decoding:

Target model: Qwen3.5-9B
Tensor parallel: 1 GPU (--tp-size 1)
Attention backend: triton
Spec config: --spec-topk 1, --spec-num-steps ∈ {3, 7, 15}
Benchmarks: MT-Bench, GSM8K, HumanEval (80 prompts each)
Generation: max_tokens=512, temperatures 0.0 and 1.0

Each cell reports accept length / speedup (Eagle3 throughput ÷ baseline throughput). The first column is the mean across the three datasets; the remaining columns are per-dataset results.

Temperature = 0.0

`spec-num-steps`	Mean	MT-Bench	GSM8K	HumanEval
3	2.97/2.25	2.45/1.92	3.26/2.43	3.18/2.39
7	3.97/2.59	2.80/2.00	4.62/2.90	4.48/2.88
15	4.34/2.19	2.84/1.59	5.08/2.41	5.09/2.56

Temperature = 1.0

`spec-num-steps`	Mean	MT-Bench	GSM8K	HumanEval
3	2.34/1.74	2.04/1.54	2.50/1.85	2.48/1.83
7	2.75/1.80	2.20/1.48	2.98/1.97	3.07/1.94
15	2.80/1.43	2.20/1.19	2.89/1.48	3.32/1.63

Usage (SGLang)

bash

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-9B \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path <path-to-this-repo> \
  --speculative-num-steps 7 \
  --tp-size 1 \

Adjust --speculative-num-steps based on your latency/throughput trade-off (see table above).

yuyijiong/Qwen3.5-9B-Eagle3

Qwen3.5-9B EAGLE3 Draft Model (Open-PerfectBlend, Decontaminated)