Qwen3.5-27B-DFlash — GGUF (Q4_K_M)
Q4_K_M llama.cpp quantization of
z-lab/Qwen3.5-27B-DFlash, the block-diffusion drafter for DFlash speculative decoding. Pair it with any Qwen3.5-27B target model (e.g.
Qwen/Qwen3.5-27B).
Only Q4_K_M is published here — on our RTX 3090 benchmarks Q4_K_M was the optimal operating point for this drafter; Q8_0 and F16 did not produce better end-to-end decode throughput, so they're intentionally omitted to keep the pairing simple.
Requirements
DFlash speculative decoding is not yet in upstream llama.cpp. You need the fork that carries the tape-replay rollback, hidden-state capture, and tree-aware SSM kernels:
- Fork:
spiritbuun/buun-llama-cpp (branch master)
- Built with:
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
Usage
llama-server
1./build/bin/llama-server \
2 -m /path/to/Qwen3.5-27B-target.Q4_K_M.gguf \
3 -md /path/to/dflash-draft-q4_k_m.gguf \
4 --spec-type dflash \
5 -ngl 99 -ngld 99 \
6 -np 1 -c 6048 -cd 256 \
7 -fa on -b 256 -ub 64 \
8 --host 0.0.0.0 --port 8080 --jinja
llama-speculative-simple
1./build/bin/llama-speculative-simple \
2 -m /path/to/Qwen3.5-27B-target.Q4_K_M.gguf \
3 -md /path/to/dflash-draft-q4_k_m.gguf \
4 --spec-type dflash \
5 -ngl 99 -ngld 99 \
6 -c 4096 --draft-max 16 --draft-min 1 \
7 -p "Write a Python mergesort."
Observed performance (RTX 3090, Qwen3.5-27B-heretic target Q4_K_M)
| Workload | Draft ON (t/s) | Draft OFF (t/s) | Acceptance |
|---|
| Code (raw prompt) | 140 | 40 | 69 % |
| Code (chat template) | 99 | 40 | 37 % |
| Prose (raw prompt) | 60 | 40 | 77 % |
Heads-up on chat templates. The drafter was trained on raw continuations; wrapping the prompt in Qwen chat-template tokens (<|im_start|> etc.) shifts the hidden-state distribution the drafter's cross-attention expects. Acceptance on code drops from 69 % → 37 % when the template is applied, and end-to-end throughput drops correspondingly. Raw-mode generation is still a strong speedup; chat-mode is a more modest one. A drafter retrained on chat-formatted data would be needed to recover the full speedup under a chat template.
Quantization details
- Source:
z-lab/Qwen3.5-27B-DFlash (BF16 safetensors, 2 B parameters)
- Converter:
convert_hf_to_gguf.py from spiritbuun/buun-llama-cpp (adds DFlashDraftModel arch)
- Quant:
llama-quantize → Q4_K_M
- File size: ~1.0 GB
- Tensors: drafter transformer + projection heads + cross-attention layers targeting Qwen3.5-27B layer ids
[1, 16, 31, 46, 61]
Reproducing the conversion
Heads-up (tokenizer error): the upstream z-lab/Qwen3.5-27B-DFlash repo ships only config.json, model.safetensors, dflash.py, and a README — no tokenizer files. The drafter shares the target model's tokenizer. If you run convert_hf_to_gguf.py directly against the z-lab repo you'll get a vocab / tokenizer error. Copy the Qwen3.5 tokenizer files into the drafter directory first.
1# 1. Pull the DFlash drafter weights
2hf download z-lab/Qwen3.5-27B-DFlash --local-dir ./dflash-drafter
3
4# 2. Pull tokenizer files from the target model into the same directory
5hf download Qwen/Qwen3.5-27B \
6 tokenizer.json tokenizer_config.json vocab.json merges.txt \
7 special_tokens_map.json \
8 --local-dir ./dflash-drafter
9
10# 3. Convert to GGUF (BF16/F16 first, then quantize)
11python convert_hf_to_gguf.py ./dflash-drafter \
12 --outtype f16 \
13 --outfile dflash-draft-f16.gguf
14
15# 4. Quantize to Q4_K_M
16./build/bin/llama-quantize dflash-draft-f16.gguf dflash-draft-q4_k_m.gguf Q4_K_M
Required files in ./dflash-drafter/ before step 3:
| File | Source |
|---|
config.json | z-lab/Qwen3.5-27B-DFlash (has architectures: ["DFlashDraftModel"]) |
model.safetensors | z-lab/Qwen3.5-27B-DFlash |
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, special_tokens_map.json | Qwen/Qwen3.5-27B |
No special converter flag is needed — convert_hf_to_gguf.py auto-detects the DFlashDraftModel architecture from config.json and registers the correct tensor mappings + GGUF hparams (dflash.block_size, dflash.mask_token_id, dflash.target_layer_ids, dflash.n_target_features).
Original model card — z-lab/Qwen3.5-27B-DFlash
Reproduced from the upstream model page. License: MIT.
Overview
Qwen3.5-27B-DFlash is a lightweight drafter component for DFlash speculative decoding. It must be used with the target model Qwen/Qwen3.5-27B.
What is DFlash?
DFlash is a novel speculative decoding method using a lightweight block diffusion model for drafting, enabling efficient, high-quality parallel drafting that significantly speeds up inference.
Upstream Quick Start (vLLM / SGLang)
vLLM
1uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
2
3vllm serve Qwen/Qwen3.5-27B \
4 --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15}' \
5 --attention-backend flash_attn \
6 --max-num-batched-tokens 32768
SGLang
1uv pip install "git+https://github.com/sgl-project/sglang.git@refs/pull/20547/head#subdirectory=python"
2
3python -m sglang.launch_server \
4 --model-path Qwen/Qwen3.5-27B \
5 --speculative-algorithm DFLASH \
6 --speculative-draft-model-path z-lab/Qwen3.5-27B-DFlash \
7 --speculative-num-draft-tokens 16 \
8 --tp-size 1 \
9 --attention-backend fa3 \
10 --mem-fraction-static 0.75 \
11 --trust-remote-code
Upstream Benchmarks (NVIDIA B200, BF16)
Up to 5.2× speedup over autoregressive baseline at concurrency 1 on HumanEval with block size 16.
| Task | Concurrency | AR | DFlash (B16) | Speedup |
|---|
| Math500 | 1 | 84 | 397 | 4.7× |
| HumanEval | 1 | 83 | 427 | 5.2× |
| GSM8K | 1 | 83 | 330 | 4.0× |
Citation
1@article{chen2026dflash,
2 title = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
3 author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
4 journal = {arXiv preprint arXiv:2602.06036},
5 year = {2026}
6}
License
MIT — inherited from the upstream model. This repository redistributes a quantized derivative under the same terms.
