A weight-only int8 version of z-lab/Qwen3.5-4B-DFlash,
the block-diffusion DFlash drafter for Qwen3.5-4B. Architecture, vocab, and behavior are unchanged
from the upstream checkpoint; only the Linear layer weights are quantized to int8.
Produced via llm-compressor 0.11.0
(QuantizationModifier oneshot, no calibration data needed — pure RTN on the weights).
Validation: this checkpoint vs upstream bf16
Tested as the DFlash drafter for a paroquant w4a16-quantizedQwen3.5-4B
target in vLLM 0.22.1 (num_speculative_tokens=15, greedy, 20 prompts / 60% no-think
ultrachat-style + 40% gsm8k-style; total ~425 draft steps each).
upstream bf16
W8A16 (this)
Δ
Mean accept length
3.60 tok / step
3.56 tok / step
-0.04
Per-token accept rate
24.03%
23.74%
-0.29 pp
End-to-end throughput
1.41 req/s
1.39 req/s
-1.4%
Per-position acceptance (positions 0–14 of the 15 spec tokens):
pos
bf16
W8A16
Δ
0
86.8%
87.1%
+0.3
1
67.2%
67.3%
+0.1
2
47.6%
48.1%
+0.5
3
32.1%
32.5%
+0.4
4
25.2%
25.7%
+0.5
5
22.2%
22.4%
+0.2
6
16.5%
16.8%
+0.3
7
11.8%
12.1%
+0.3
8
10.8%
11.2%
+0.4
9
9.9%
10.0%
+0.1
10
9.0%
9.1%
+0.1
11
7.5%
7.7%
+0.2
12
7.1%
7.0%
-0.1
13
6.1%
6.1%
0
14
5.7%
5.6%
-0.1
All differences are within sampling noise for 425 trials. Treat the int8 drafter as
lossless for practical purposes.
The chain-throughput gain on this hardware (RTX 3090) was negligible because the
drafter is only ~30% of the spec-decode chain wall-clock — the target verifier
dominates. The win here is memory (~480 MB of GPU returned), which matters
mostly when stacking multiple speculators on the same device or pushing
longer max_model_len / larger batch.
Usage
vLLM (recommended)
The drafter pairs with any Qwen3.5-4B (or paroquant'd Qwen3.5-4B) target.
vLLM 0.22.1's qwen3_dflash.py reads qkv_proj.weight / fc.weight directly via
F.linear. This bypasses the quant-aware Linear.forward and trips on any quantized
drafter (compressed-tensors, bnb, paroquant, …). The patch covers four sites and
keeps the fused-KV fast path active even for quantized weights — no per-layer
fallback at runtime:
Defer _build_fused_kv_buffers() out of load_weights. The eager call
runs beforeprocess_weights_after_loading and so before Marlin / etc. have
set up their internals (e.g. g_idx_sort_indices). Move the build to the lazy
path in precompute_and_store_context_kv, which fires after warmup.
_build_fused_kv_buffers — detect quantized qkv_proj. For the FP case,
keep the original weight[q_size:] slice + cat. For the quantized case, recover
the effective KV-projection weight by running an identity matrix through each
layer's qkv_proj(...), slicing the K+V output rows, and concatenating into one
bf16 fused tensor. This works for any quant scheme without needing knowledge
of its internal storage. ~50 MB extra runtime memory (K+V slice across 5 layers).
DFlashQwen3Attention.forward — call self.qkv_proj(hidden_states) when
.weight isn't a plain FP tensor, instead of F.linear(qkv_proj.weight, ...).
This is the regular draft-pass attention; uses Marlin int8 GEMM directly.
combine_hidden_states — read fc.weight_scale.dtype instead of
fc.weight.dtype to pick the activation cast dtype when fc is quantized.
The first time you load this checkpoint with the patch in place, vLLM logs:
DFlash drafter: qkv_proj is quantized; extracting fused KV weight via identity
probe (dtype=torch.float16, hidden=2560). Restores the fast fused-GEMM path in
context KV precompute.
A reference implementation of the three-site patch lives in the
discussion thread
on this repo, and we plan to submit it to vLLM upstream.
