You can deploy this model and start earning money today!

stevenabreu7/
gpt-oss-120b-speculator.eagle3

Name: gpt-oss-120b-speculator.eagle3
Availability: InStock

2503.01840

Views

No views yet

GPT-OSS-120B Eagle3 Speculator Training

This repository contains the training script for an EAGLE-3 speculative decoding speculator for openai/gpt-oss-120b.

Method

Algorithm: EAGLE-3 (Training-Time Test)
Target Model: openai/gpt-oss-120b
Training Data: HuggingFaceH4/ultrachat_200k (train_sft split)
Speculator Size: ~855M parameters
Hidden Size: 2880
Draft Vocab Size: 64000
Target Vocab Size: 201088
Learning Rate: 5e-5
Optimizer: AdamW (β1=0.9, β2=0.95)
Gradient Clipping: 1.0
Scheduler: Linear with warmup
Loss Function: KL divergence on token distributions with training-time test steps

Architecture Details

Following the reference RedHatAI/gpt-oss-120b-speculator.eagle3:

Single-layer Llama decoder (hidden_size=2880, intermediate_size=2880)
64 attention heads, 8 key-value heads
Multi-layer feature fusion (3× hidden_size → hidden_size)
norm_before_fc=True, norm_before_residual=True
embed_requires_grad=False
max_position_embeddings=131072

Pipeline

The training follows the offline pipeline from vllm-project/speculators:

prepare_data.py — Tokenize conversations, generate loss masks for assistant tokens
launch_vllm.py — Serve GPT-OSS-120B with hidden-state extraction enabled
data_generation_offline.py — Generate hidden states for all training samples via vLLM API
train.py — Train Eagle3 speculator on pre-generated hidden states
push_to_hub — Upload best checkpoint to HuggingFace Hub

Usage with vLLM

bash

vllm serve openai/gpt-oss-120b \
  --tensor-parallel-size 4 \
  --speculative-config '{
    "model": "stevenabreu7/gpt-oss-120b-speculator.eagle3",
    "num_speculative_tokens": 5,
    "method": "eagle3"
  }' \
  --no-enable-prefix-caching \
  --max-num-seqs 64 \
  --enforce-eager

References

EAGLE-3 Paper: Li et al., 2025
vllm-project/speculators: GitHub
Speculators Docs: docs.vllm.ai