This repository contains the epoch 3 (zero-indexed), global step 70000 snapshot from an online EAGLE3 draft-model training run for Qwen/Qwen3-4B-Instruct-2507.

This is a speculative-decoding draft model. It is intended to be used together with the target model and is not a standalone chat or text-generation model.

Part of the Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT Checkpoints collection.

Checkpoint

Field	Value
Target model	`Qwen/Qwen3-4B-Instruct-2507`
Training method	Online EAGLE3 with SpecForge
Epoch	`3` (zero-indexed)
Global step	`70000` / `231810`
Draft parameters	`202,700,416`
Draft layers	`1` decoder layer
Precision	`bfloat16`
Sliding-window size	`256` tokens

Training Parameters

The values below were recovered from the saved training state and checkpoint config, rather than inferred only from the directory name.

Parameter	Value
Training data	Cleaned ShareGPT JSONL
Raw input records	`92,724` before preprocessing
Chat template	`qwen`
Maximum sequence length	`2048`
Epochs	`10`
Total optimizer steps	`231810`
Per-device batch size	`1`
Data-parallel size	`4`
Effective global batch size	`4`
Gradient accumulation	`1`
Target-model tensor parallel size	`1`
Optimizer	AdamW with FP32 master parameters
Peak learning rate	`1e-4`
LR schedule	Cosine annealing with linear warmup
Warmup ratio	`0.015`
Weight decay	`0.0`
Maximum gradient norm	`0.5`
EAGLE3 TTT length	`7`
Draft attention backend	`sdpa`
Target-model backend	SGLang (`flashinfer`)
Draft sliding window	`256`
Seed	`0`
Dataset preprocessing workers	`64`
Dataloader workers	`4`
Save interval	`5000` steps
Log interval	`50` steps
SpecForge source revision	`9fbbde8ab5d6ee69fb0af3701330027b8beca37a`
Transformers version recorded in config	`4.57.1`

The training run did not configure an evaluation dataset or experiment tracker. No validation, acceptance-rate, or throughput metrics are reported for this checkpoint. The latest checkpoint is not necessarily the best checkpoint for every workload; compare checkpoints on representative prompts before deployment.

Architecture

The draft uses LlamaForCausalLMEagle3 with:

hidden size 2560;
intermediate size 9728;
32 attention heads and 8 key/value heads;
target vocabulary size 151,936 and draft vocabulary size 32,000;
one decoder layer and causal sliding-window attention of 256 tokens.

Usage with SGLang

bash

python3 -m sglang.launch_server \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path huluhuluu/Qwen3-4B-Instruct-2507-EAGLE3-ShareGPT-epoch-3-step-70000 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

The speculative tree settings above are starting values, not benchmark-derived recommendations for this checkpoint. Tune them for the serving workload and SGLang version.

Files

model.safetensors: EAGLE3 draft weights.
config.json: draft architecture and sliding-window configuration.
training_state.pt: optimizer, scheduler, epoch, global step, and training arguments for resuming training.

training_state.pt uses PyTorch pickle-based serialization and is not required for inference. Only deserialize it in a trusted environment. Prefer model.safetensors for inference.

Limitations

This checkpoint must be paired with the exact target-model family used for training.
It was trained on a cleaned local ShareGPT file; no public dataset revision was recorded.
No benchmark or safety evaluation artifacts were saved with the run.
Speculative-decoding gains depend on prompt distribution, generation settings, hardware, and serving software.

Licenses

The weights are published under Apache-2.0, following the target model. The SpecForge training code is MIT licensed. Users are responsible for reviewing the terms of the target model and training data for their use case.