1.90× faster decode on code editing than the stock NVFP4 checkpoint, on a single
NVIDIA DGX Spark (GB10, 128 GB unified memory, ~273 GB/s).
Measured, not estimated: 72.9 → 138.7 tok/s single-stream decode on a code-edit
workload.
This repo ships everything needed to reproduce that number: the quantized weights, the
speculative-decoding draft head, the exact vLLM flags, and the deployment manifest. See
Why this repo is self-contained — that section exists
because of a real failure mode we hit.
lm_head quantized to int4, group 32, symmetric, no zero-point
model.safetensors.index.json
tensor → shard map. Required: lm_head lives in its own shard
eagle3-head/
EAGLE3 draft head, RoPE-rescaled to 262144 (see below)
deploy/vllm-deployment.yaml
the Kubernetes Deployment that produced the measured numbers
config.json, chat_template.jinja, tokenizer files
unchanged from the base checkpoint
Measured performance
Single-stream decode, steady state (prefill excluded from the denominator). Runs where any
other process was using the GPU were discarded — the node exposes 2 time-sliced GPU
replicas over 1 physical GB10, so a neighbouring pod silently halves your throughput.
config
code_edit
free_gen
long_ctx (52K prompt)
stock NVFP4, no speculation
72.9
71.4
42.3
+ EAGLE3 k=3
131.2 (1.80×)
52.8 (0.74×)
55.0 (1.30×)
+ EAGLE3 k=3 + int4 lm_head
138.7 (1.90×)
—
—
Acceptance rate with EAGLE3 k=3:
workload
tokens/step
acceptance
code_edit
3.89 / 4.0
96 %
long_ctx
2.00
33 %
free_gen
1.46
15 %
Read this before enabling EAGLE3
EAGLE3 makes free-form generation 26 % SLOWER (71.4 → 52.8 tok/s). At 15 % acceptance
you pay for the draft forward and get nothing back. The head was trained on code
(OpenCoder-LLM/opc-sft-stage1), so it is excellent at code editing and poor at prose.
Enable it if your traffic is agentic coding. Do not enable it for a general chat endpoint.
ngram / prompt-lookup is a trap here
We measured it: 80 % acceptance, 3.39 tokens/step — and 45 % SLOWER than no speculation
at all (72.9 → 39.9 tok/s). The reason is in vLLM's source: NgramProposer sets
num_numba_thread_available = min(1, cpu_count // 2), hard-capped at 1, so the KMP search
runs single-threaded on CPU at every decode step. EAGLE3 drafts on the GPU, which is why it
converts its acceptance into real throughput and ngram does not.
Notes that cost us time, so they are written down:
262144 is native.max_position_embeddings=262144, rope_scaling=null. No YaRN needed,
and VLLM_ALLOW_LONG_MAX_MODEL_LEN is deliberately NOT set so a real length error stays loud.
The EAGLE3 head adds one layer on top of the target's 48, so it needs ~2 % more KV cache
than the same context without speculation. At 262144/fp8 that is 12.25 GiB, not 12.0. A
budget that fits without the drafter will fail with it.
No --quantization flag. vLLM autodetects compressed-tensors from config.json.
No --reasoning-parser. The Instruct variant never emits <think>.
The lm_head shard loads through MarlinLinearKernel (CompressedTensorsWNA16).
That change is not cosmetic. vLLM builds the draft's rotary cache with
torch.arange(max_position_embeddings), so with the original config every position ≥2048
indexes out of bounds. RoPE is analytic, so extending the cache just computes more rows at
the same theta.
Known, unverified: the head declares rope_theta=1e6 while the target declares 1e7. It
works and accepts at 96 % on code edits; whether matching them would improve long-context
acceptance (currently 33 %) has not been tested.
How the int4 lm_head was made
The stock NVFP4 checkpoint leaves lm_head in BF16 (it is listed in quantization_config.ignore).
At vocab 151936 × hidden 2048 that is 622 MB read per decoded token. Decode on GB10 is
memory-bandwidth bound, so that is pure tax.
Quantized to int4/group-32 it becomes 175 MB — 447 MB less per token.
Symmetric, no zero-point, values in [-8, 7]. Quantization error: L2 relative 9.19 %,
max abs 0.02344.
Cost: 79 seconds, no GPU required. Measured gain: +5.7 % on top of EAGLE3 (less than the
~15-20 % the bandwidth math predicts, because with speculation the bottleneck shifts — the
draft head carries its own lm_head).
Full quantization from BF16 was evaluated and rejected: field-by-field, the public NVFP4
checkpoint already uses an equivalent observer, the same effective scale_dtype, and a code
calibration set. Reproducing it would be 4-8 GPU-hours for no measurable gain. The lm_head
is the only material difference.
Why this repo is self-contained
We tried to reproduce a published 154 tok/s figure for a different model from its public
repo and could not. The reason turned out to be that the public repo shipped only the base
weights — the speculative-decoding module that produced the speedup, the split lm_head
shard, and the model.safetensors.index.json that maps them were never uploaded. Anyone
downloading it gets a checkpoint with no speculation and roughly a third of the tokens per
forward pass, with nothing in the repo to indicate why.
So this repo ships the draft head, the index, the exact flags, the container build, and the
measurement methodology including what we discarded and why. If a number here does not
reproduce, that is a bug and we want to know.
Reproducing the measurements
The benchmark harness gates every run on /metrics and throws away any sample taken while
another request — or another pod on the same time-sliced GPU — was active. Without that
gate the same config measures anywhere from 27 to 139 tok/s, and the contaminated numbers
look perfectly plausible.
State the workload with any tok/s claim. On this model the same config spans 52.8 → 138.7
tok/s depending only on whether the task is prose or code editing.
License
Apache-2.0, inherited from Qwen/Qwen3-Coder-30B-A3B-Instruct.
eagle3-head/ is MIT, from lmsys.
Full benchmark campaign
Speed, tool-calling and agentic results for this model alongside three others on the same
DGX Spark — including two optimizations that were measured and rejected — with all raw
evidence:
