Qwen3.5-397B-A17B-DFlash
This is a joint release collaboration between
Z-Lab,
Modal, and
SGLang. The draft model is mirrored across the following Hugging Face repositories:
This repository contains a DFlash draft model for Qwen/Qwen3.5-397B-A17B. It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server.
DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution.
Quick Start
Installation
DFlash serving requires a recent SGLang build. We recommend installing SGLang from a commit at or after ec36dde58083aca8f26c3740332498a11a06debf, which includes DFlash v2 overlap scheduling, DFlash sliding-window draft attention, and the required FlashInfer integration.
1pip install -U \
2 "git+https://github.com/sgl-project/sglang.git@ec36dde58083aca8f26c3740332498a11a06debf#subdirectory=python" \
3 transformers hf_xet
For production serving, use a CUDA image with Blackwell support, such as lmsysorg/sglang:v0.5.13-cu130, and install the pinned SGLang commit inside the image.
Launch Server
This model should be used with an inference server that supports DFlash speculative decoding. An example SGLang deployment is:
1export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
2
3python -m sglang.launch_server \
4 --model-path Qwen/Qwen3.5-397B-A17B \
5 --trust-remote-code \
6 --speculative-algorithm DFLASH \
7 --speculative-draft-model-path z-lab/Qwen3.5-397B-A17B-DFlash \
8 --speculative-dflash-block-size 8 \
9 --speculative-draft-attention-backend fa4 \
10 --attention-backend trtllm_mha \
11 --linear-attn-prefill-backend triton \
12 --linear-attn-decode-backend flashinfer \
13 --mamba-scheduler-strategy extra_buffer \
14 --tp-size 8 \
15 --max-running-requests 32 \
16 --cuda-graph-max-bs-decode 32 \
17 --cuda-graph-backend-prefill tc_piecewise \
18 --enable-flashinfer-allreduce-fusion \
19 --mem-fraction-static 0.8 \
20 --host 0.0.0.0 \
21 --port 30000
Block size 8 is the recommended default for higher-concurrency serving. Block size 16 gives longer accept lengths and the best concurrency-1 throughput in most workloads.
Benchmark Results
We benchmarked DFlash against the autoregressive baseline and Qwen3.5's built-in MTP draft path. DFlash reaches up to 4.31x speedup at concurrency 1 and 2.77x at concurrency 32. Across the benchmark suite, DFlash delivers higher throughput than MTP at every tested setting.
Setup
- Runtime: SGLang on 8x NVIDIA B200 GPUs, tensor parallel size 8,
bfloat16
- Backends:
trtllm_mha target attention, fa4 DFlash draft attention, flashinfer linear attention
- Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with the Qwen chat template
- Decoding: greedy, thinking enabled, max output length 4096 tokens
- Measurement: 5 independent runs per configuration at concurrency 1 and 32 with continuous batching
- Throughput: generated output tokens / wall-clock benchmark time, including prefill and scheduling
- Accept length:
completion_tokens / spec_verify_ct per generation turn, averaged across generation turns
For reproduction, the evaluation scripts, Modal runner, runtime patches, and raw benchmark outputs are included under
benchmark/.
Throughput and Speedup
Each cell is output tok/s (speedup). Bold marks the fastest speculative configuration in each row.
Concurrency 1
| Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 |
|---|
| gsm8k | 204.6 (1.00x) | 484.5 (2.37x) | 530.0 (2.59x) | 536.7 (2.62x) | 689.0 (3.37x) | 439.2 (2.15x) | 711.3 (3.48x) |
| math500 | 204.4 (1.00x) | 505.2 (2.47x) | 551.3 (2.70x) | 589.4 (2.88x) | 762.6 (3.73x) | 499.8 (2.44x) | 831.9 (4.07x) |
| humaneval | 202.9 (1.00x) | 483.0 (2.38x) | 543.6 (2.68x) | 557.9 (2.75x) | 752.5 (3.71x) | 480.5 (2.37x) | 874.6 (4.31x) |
| mbpp | 204.6 (1.00x) | 487.4 (2.38x) | 550.3 (2.69x) | 543.7 (2.66x) | 751.2 (3.67x) | 445.6 (2.18x) | 807.8 (3.95x) |
| mt-bench | 202.6 (1.00x) | 442.9 (2.19x) | 473.6 (2.34x) | 441.5 (2.18x) | 545.9 (2.69x) | 338.0 (1.67x) | 515.3 (2.54x) |
Concurrency 32
| Workload | Baseline | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 |
|---|
| gsm8k | 2597.7 (1.00x) | 4462.3 (1.72x) | 5078.0 (1.95x) | 4902.9 (1.89x) | 6158.9 (2.37x) | 4079.5 (1.57x) | 5648.8 (2.17x) |
| math500 | 2615.5 (1.00x) | 4838.6 (1.85x) | 5318.6 (2.03x) | 5601.6 (2.14x) | 6910.5 (2.64x) | 4855.0 (1.86x) | 6857.8 (2.62x) |
| humaneval | 2452.7 (1.00x) | 3838.1 (1.56x) | 5081.4 (2.07x) | 4347.4 (1.77x) | 6666.0 (2.72x) | 3917.2 (1.60x) | 6783.7 (2.77x) |
| mbpp | 2582.3 (1.00x) | 4129.8 (1.60x) | 5265.7 (2.04x) | 4461.0 (1.73x) | 6681.4 (2.59x) | 3783.7 (1.47x) | 6344.1 (2.46x) |
| mt-bench | 2537.3 (1.00x) | 4024.6 (1.59x) | 4425.3 (1.74x) | 3991.7 (1.57x) | 4763.2 (1.88x) | 3100.3 (1.22x) | 4016.0 (1.58x) |
Accept Length
Mean accept length at concurrency 1. Bold marks the higher value in each matched MTP/DFlash pair.
| Workload | MTP steps=3 | DFlash block=4 | MTP steps=7 | DFlash block=8 | MTP steps=15 | DFlash block=16 |
|---|
| gsm8k | 3.578 | 3.548 | 5.612 | 5.738 | 7.006 | 7.696 |
| math500 | 3.636 | 3.622 | 5.810 | 6.003 | 7.294 | 8.212 |
| humaneval | 3.643 | 3.697 | 5.893 | 6.292 | 7.541 | 9.344 |
| mbpp | 3.532 | 3.588 | 5.357 | 5.860 | 6.398 | 7.801 |
| mt-bench | 3.244 | 3.188 | 4.567 | 4.584 | 5.256 | 5.523 |
Citation
If you find DFlash useful, please cite the original paper:
1@article{chen2026dflash,
2 title = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
3 author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
4 journal = {arXiv preprint arXiv:2602.06036},
5 year = {2026}
6}
