Qwen3.8-27B-DFlash2-FP8
An FP8 quantization of the DFlash2 draft model for Qwen/Qwen3.8-27B. It is not a
standalone language model: it drafts tokens inside a speculative-decoding server for a
target model to verify.
Quantizing a drafter is unusually low-risk: every token it proposes is verified against
the target, so a worse drafter costs speed, never correctness. The upside is real on a
bandwidth-bound machine — the drafter runs once per step, so its weight bytes are paid
every step.
⚠️ Requires a patched vLLM
This will not load on stock vLLM. Two separate defects sit in front of it:
- The draft model's quantization config never reaches
packed_modules_mapping, so fused
projections are built unquantized and loading dies with a confusing
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'data' —
vllm#53116,
vllm#53107,
PR #53122.
- DFlash's fused context-KV projection slices the raw
qkv_proj.weight and applies it
with a bare F.linear, bypassing quant_method — so a quantized drafter dies at
startup profiling with expected mat1 and mat2 to have the same dtype —
vllm#51581,
PRs #51620,
#51684.
Until those land, this checkpoint is mainly useful as a test artifact for the people
fixing them: reproducing either defect needs a quantized DFlash drafter.
Quantization scheme
W8A8, FP8 e4m3 weights with per-channel scales and dynamic per-token activation
scales. There is no stored input_scale; the activation scale is computed at runtime.
vLLM selects CompressedTensorsW8A8Fp8.
Quantized: self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, fc — 2-D weights only.
Deliberately left in BF16: the two-tap convolutions (*_conv), the
candidate_selector and its codebooks, and all norms. Together they are 0.39 GB — the
saving is small and they are the parts that make block-diffusion drafting work.
The exact script that produced this checkpoint is included as quantize.py.
Result: 3.85 GB → 2.25 GB.
Measurements
NVIDIA GB10 (DGX Spark, sm_121a), TP=1, target RadixArk/Qwen3.8-27B-NVFP4 (NVFP4
W4A4 head), num_speculative_tokens=7, FlashInfer, fp8 KV cache. 24 replayed real agent
contexts, max_tokens=256. Both arms on the same engine and the same session — only the
drafter differs.
| drafter | decode | acceptance length | TTFT |
|---|
BF16 (z-lab/Qwen3.8-27B-DFlash2) | 41.3 tok/s | 4.24 | 5.6 s |
| FP8 (this) | 43.8 tok/s | 4.24 | 5.6 s |
+6.1 % decode at an unchanged acceptance length. The quantization was predicted to
buy ~7.5 % from the byte budget alone (the drafter is 15.5 % of the bytes moved per step
on a machine measured at 231.8 of 273 GB/s); the measured 6.1 % is close, and the
unchanged acceptance length says the drafts themselves did not get worse.
Scope of these numbers: one target, one harness, n=24, single stream. They are not a
general claim about FP8 drafters.
Related work
lued/Qwen3.8-27B-DFlash2-W8
(2026-08-19) quantizes the same base drafter, and predates this one. It is a
different
scheme, not a competing build of the same thing:
| lued/…-W8 | this |
|---|
| weights | INT8, group-128, pack-quantized | FP8 e4m3, per-channel, float-quantized |
| activations | none (W8A16, weight-only) | dynamic per-token (W8A8) |
| built with | llm-compressor QuantizationModifier | the included quantize.py |
| size | 2.02 GiB | 2.25 GB |
The two have since been measured against each other, on the same engine and in the same
session as the numbers above — only the drafter was swapped:
| drafter | decode | acceptance length | TTFT |
|---|
BF16 (z-lab/Qwen3.8-27B-DFlash2) | 41.3 tok/s | 4.24 | 5.6 s |
INT8 W8A16 (lued/…-W8) | 43.2 tok/s | 4.16 | 5.6 s |
| FP8 W8A8 (this) | 43.8 tok/s | 4.24 | 5.6 s |
The expectation that FP8's native Blackwell support would beat Marlin-routed INT8 on GB10
was not confirmed. FP8 is 1.4 % ahead of INT8, which is the same order as the ~2.5 %
session-to-session drift we have measured elsewhere on this box: at this sample size the
two schemes are not distinguishable. Pick either.
What the run does support is the other half: both quantized drafters land roughly 5 %
above BF16 — 4.6 % for INT8, 6.1 % for FP8 — at a practically unchanged acceptance
length (4.16 and 4.24 against BF16's 4.24). That is the result worth carrying over.
Same scope limit as above: n=24, single stream, one target, one harness. Their card
reports acceptance unchanged against BF16 as well, from a different harness.
Credits
Base model:
z-lab/Qwen3.8-27B-DFlash2,
itself a mirror of
incoai/Qwen3.8-27B-DFlash2.
DFlash2:
blog ·
code.
Apache-2.0, inherited from the base model.
