Mixed-precision NVFP4 via NVIDIA ModelOpt: MLP and attention output projections at 4 bits, everything precision-sensitive left in BF16.
Retrained so the drafter sees what the model actually serves. The previous release was trained on a corpus with two mismatches against inference: tool examples used Hermes-style JSON (<tool_call>{"name":...}) while serving emits XML (<function=…><parameter=…>), and prompts were rendered with a generic template carrying no think blocks, no reasoning-effort preamble and no tools preamble. Both are now closed — every response is regenerated on-policy through the published chat template.
Previous release
This model
Δ
Agentic tool calling
3.299
3.766
+14.2%
Held-out accept length
2.761
2.886
+4.5%
Throughput
150.74 tok/s
155.75 tok/s
+3.3%
Checkpoint size
1.30 GB
1.30 GB
—
Same size, same recipe, same behaviour — the gain is purely in how often the target accepts a drafted block.
1.91× the decode throughput of the same checkpoint without speculation.
Throughput for the top two rows was measured cold in one session; the older rows come from earlier sessions, where run-to-run variance on this harness is roughly 2%.
Per domain
Acceptance is measured per request with the prefix cache flushed between requests, derived from cumulative verify counters rather than a sampled gauge.
Domain
Previous
This model
Δ
Math
3.810
4.388
+15.2%
Coding
3.729
3.804
+2.0%
JSON / structured output
3.328
3.571
+7.3%
Chat
2.305
2.450
+6.3%
Long-context
2.484
2.334
−6.1%
Instruction
2.195
2.323
+5.9%
Overall
2.761
2.886
+4.5%
Long-context is the one regression. The training corpus caps sequences at 2,048 tokens, so it was never well represented, and the shift toward agentic and think-block data appears to have cost it.
Agentic tool calling
Real tool use — tools declared, <tool_call> XML emitted, tool responses replayed — measured on 60 scenarios across 10 tool schemas. This is the workload the retraining targeted, and it improved in every phase:
Phase
Previous
This model
Δ
Parallel calls
4.169
5.075
+21.7%
Initial call
3.683
4.302
+16.8%
Error recovery
3.028
3.493
+15.4%
After a tool result
3.524
3.895
+10.5%
Second call in a loop
3.155
3.445
+9.2%
Two-step chain
2.605
2.816
+8.1%
Overall
3.299
3.766
+14.2%
Tool-call emission rate is unchanged (38/60 on both), so this is an acceptance gain rather than a change in what the model decides to do.
Note that the "JSON / structured output" row above is schema-constrained generation, not tool calling — those held-out prompts declare no tools. It was previously labelled "Tool calling", which understated the drafter on the workload it is named after.
Quantization recipe
Mixed precision, chosen by measuring every stage rather than quantizing everything:
Q/K/V are deliberately not quantized. A full-NVFP4 variant including them was built and measured: it reaches 150.73 tok/s but acceptance falls to 2.546. Keeping QKV in BF16 costs ~3 tok/s and buys back the acceptance, which is the better production trade.
Measured ladder:
Variant
tok/s
Accept
VRAM
MLP only
147.61
2.521
1.58 GB
MLP + O (this model)
147.87
2.792
1.41 GB
MLP + O + QKV
150.73
2.546
1.17 GB
Adding the O projection to MLP recovers acceptance fully; adding QKV loses it again.
fc (the 25600→5120 aux projection) stays BF16 because SGLang constructs it as a plain nn.Linear with no quantized loader.
Calibration
Post-training quantization with NVIDIA ModelOpt, calibrated to resemble real serving rather than synthetic input:
512 conversations, regenerated on-policy by the NVFP4 target at temperature 0
Tool-weighted mixture (37.5% tool calling) to protect the most structurally rigid output
Real captured target auxiliary features and real token embeddings for the draft input path
Calibration data is disjoint from the evaluation set — no benchmark prompt was ever calibrated on
The tool weighting is measurable: it lifts tool acceptance from 3.15 to 3.85 versus a balanced calibration, at a cost of 0.8 tok/s overall.
Usage
Requires the Qwen3.8 SGLang build. Pin the image — DSpark runtime support moves quickly.
--mamba-ssm-dtype bfloat16 — without it the hybrid GDN state cache cannot allocate a single request slot and the server exits at startup.
--max-mamba-cache-size 8 — this must leave headroom. Under DSpark, Qwen3.8 consumes 4 GDN state slots per request, so --max-mamba-cache-size 4 provisions exactly one request and nothing spare. Radix-cached prefixes hold slots from the same pool under extra_buffer_lazy, so if a cached state is locked when the next request needs a slot, eviction fails and the server aborts with AssertionError: Can not alloc mamba cache. Size 8 halves peak utilisation (0.75 → 0.38 under abort-heavy multi-turn traffic) at a cost of some KV pool. Rule of thumb: --max-mamba-cache-size ≥ 4 × --max-running-requests + 4.
--speculative-draft-model-quantization modelopt_fp4 — this checkpoint is pre-quantized; loading it any other way will fail.
--mem-fraction-static 0.86 — leave ~3.5 GB free. See below.
Context limits
The freed 1.31 GB converts directly into KV pool: 122,995 tokens at 64K settings, versus 72,485 for the BF16 drafter.
max_total_num_tokens is not the serving limit
At --mem-fraction-static 0.94 the server starts, reports a large pool, and then dies on the first long request:
causal_conv1d_triton.py:443 out = torch.empty_like(x)
torch.OutOfMemoryError: Tried to allocate 40.00 MiB. 9.81 MiB is free.
The pool is allocated optimistically and does not reserve for GDN prefill activations. Leave ~3.5 GB free and it is stable.
Concurrency and the 262K question
SGLang caps this configuration at max_running_requests = 1. Qwen3.8's GDN state cache costs ~147 MB per request across 4 slots; there is no room for a second on 32 GB. This ceiling is inherent to the target's hybrid architecture — the built-in MTP head hits it identically.
With this drafter
Target alone, no speculation
Max context
~116K+
262K (verified: 252,849-token prompt served)
Short-prompt decode
147.87 tok/s
81.58 tok/s
Cold TTFT at 250K
n/a
~121 s
Full 262K with speculation does not fit on one 32 GB card even at 4 bits: the target's own FP8 KV needs ~8 GiB at 262K, and the draft KV pool is sized to the full target context. Use the no-speculation profile for maximum context.
Note also that speculative decoding accelerates decode only, never prefill.
Output quality is unchanged. The target verifies every drafted token under strict acceptance (speculative_accept_threshold_single=1.0, rejection sampling off). Speculation changes speed, not outputs — and that guarantee is what makes 4-bit drafting safe.
Model
Item
Value
Architecture
Qwen3DSparkModel — 5 full-attention layers
Hidden size
5120 · 40 heads · 8 KV heads · intermediate 10240
Block size (gamma)
7
Target aux layers
[4, 16, 28, 40, 52]
Markov head
vanilla, rank 256, confidence head enabled
Quantization
NVFP4 W4 group-16 (ModelOpt 0.45.0), mixed with BF16
Checkpoint
1.30 GB
Certified stack
DSpark runtime support is evolving quickly; this checkpoint was measured against exactly:
Evaluated on 240 held-out prompts (40 per domain), unseen during training and calibration. Greedy decoding, thinking disabled, 256 max new tokens. All profiles ran on identical prompts minutes apart on the same machine.
Caveat on margins. The acceptance figures average over many verification steps and are the more reliable signal. Throughput differences of 1–2% are measured across 72 requests per profile and should be read as directional.
License
Apache 2.0, matching the base drafter and the Qwen3.8-27B target.
