Steps: 49,608
Wall-clock time: 4 hours 24 minutes
Hardware: NVIDIA RTX 5090 (sm_120, 32 GB VRAM)
Method: EAGLE-3 (Li et al., 2025) self-distillation from the target model
Self-distilled training data: 50K samples generated from pyloxsystems/hochelaga-8b

Measured K-sweep (RTX 5090 sm_120, verified 2026-05-20)

Paired with pyloxsystems/hochelaga-8b-nvfp4 and Marlin NVFP4 backend (the only NVFP4 kernel that compiles for sm_120 in current public vLLM):

K (spec tokens)	1p per-stream	50p agg	100p agg	400p agg
K=1 (optimal)	197.1	2,438.1	3,844.4	4,443.2
K=2	178.6	2,028.8	2,764.5	—
K=4	154.2	1,578.1	1,890.6	—
K=8	115.7	994.6	1,136.7	—
(NVFP4 baseline, no spec, reference)	211.9	—	6,551.9	8,760.0

Findings: K=1 is the optimal speculative depth; throughput decays monotonically as K grows. On the Marlin backend specifically, the K=1 single-stream rate (197 tok/s) is slightly under the NVFP4 baseline (212 tok/s), the Marlin INT4 kernel's per-token cost absorbs the drafter's would-be latency win. The EAGLE-3 1.91× speedup observed in earlier internal measurements assumed a FlashInfer-CUTLASS FP4-native kernel that does not currently compile on sm_120.

When EAGLE-3 helps: the drafter still provides meaningful concurrency-region wins at low-to-moderate concurrency (1p-100p) and is a future-work win once FP4-native sm_120 kernels ship in stable vLLM. For now, it is documented and ready-to-deploy.

Usage with vLLM

bash

vllm serve pyloxsystems/hochelaga-8b-nvfp4 \
  --quantization nvfp4 \
  --speculative-config '{
    "model": "pyloxsystems/hochelaga-8b-eagle3-drafter",
    "num_speculative_tokens": 1,
    "method": "eagle3"
  }'

License

Apache 2.0. Inherits all licensing from the base model.

Built in Montréal · Pylox Systems Inc. · 2026