zenith1232/qwen36-eagle3-drafter-v4
EAGLE-3 speculative-decoding draft head for Qwen/Qwen3.6-35B-A3B, trained with a
Mixture-of-Experts draft block (not the usual dense draft).
Architecture
- 1 decoder block, hidden_size=2048, head_dim=256
- Full-attention self-attn (num_heads=16, num_kv_heads=2)
- MoE MLP:
- num_experts=8, top_k=2
- moe_expert_intermediate=4096
- shared_expert: True
- aux-free load balance (DeepSeek-V3 style) + router z-loss (ST-MoE)
- aux_hidden from target layers 3 / 19 / 35
- draft vocab: 32000 (shortlist of most frequent 32K target tokens)
Training
- Teacher:
Qwen/Qwen3.6-35B-A3B (bf16)
- Data: ~126M tokens of teacher trajectories (Stage B, 437750 records)
- Loss: progressive TTT (length 7, ploss weights 0.8^k) + MoE router z-loss
- Best ckpt val_ce0: 1.5411 at step 5250
Serving
Not yet supported in vanilla vLLM — the upstream Eagle3 loader assumes a
dense draft MLP. A ~100 LOC patch is needed to route through the MoE block
(in progress). Until then, load the raw safetensors via transformers and
run inference through the Eagle3MoE module in our training repo.
Files
model.safetensors — trained weights, _orig_mod. prefix strippedconfig.json — Eagle3MoE config (hidden_size, num_experts, top_k, etc)best_info.json — step + val_loss + val_ce0 of the saved checkpoint
