Qwen3.8-27B DFlash2 on 2× DGX Spark (GB10) — 135 tok/s single-stream

Mirrors: canonical repo is on GitHub at fattchris/qwen-dflash2-spark (identical content).

TL;DR quickstart

What this is

• Target model: unsloth/Qwen3.8-27B-NVFP4 (FP4 weights, FP8 attention) — the bandwidth-floor baseline on GB10.
• Draft model: z-lab/Qwen3.8-27B-DFlash2 (1.92B params, block_size 8, local convolution + candidate selector head).
• Serving stack: vLLM from PR vllm-project/vllm#52816 (z-lab/vllm-fork@19c9351, branch subsir/upstream-dflash2), compiled aarch64 in the official vllm/vllm-openai:0.27.1-aarch64 image, TP=2 across two Sparks via RoCE v2.
• The one real fix that makes the combo work: dequantize lm_head to BF16 in a copy of the NVFP4 checkpoint. Every public NVFP4 Qwen3.8 checkpoint quantizes lm_head, but DFlash2's candidate selector performs TopK over the target model's lm_head logits and hard-rejects quantized heads — so nothing out of the box works. 02-make-bf16head.sh does the surgery + config fix atomically and hardlink-safely.

Results (GB10 ×2, TP=2, NVFP4 target + 1.92B draft)

• K sweep on GB10: sweet spot is K=16 (2 draft blocks). K=24 drops acceptance (66%) and net throughput. K must be a multiple of block_size 8.
• ×5 repeat variance: 131.5–134.8 tok/s, accept 93.1% every run.
• Quality: 6/6 greedy correct vs baseline (math/Canberra/syllogism/bugfix/code) — BF16 lm_head dequant introduces zero measurable drift.
• Aggregate/concurrent throughput scales differently — C1 numbers are single-stream only, and only comparable to other C1 numbers.

Repo layout

scripts/00-download-models.sh   # hf download target + draft
scripts/01-build-image.sh       # compile z-lab vllm-fork inside official image
scripts/Dockerfile.df2          # overlay built vllm/ onto clean official image
scripts/02-make-bf16head.sh + make_bf16head.py   # the lm_head surgery
scripts/03-replicate.sh         # rsync checkpoints node1 → node2
scripts/04-launch-pair.sh       # TP2 launch, health-wait
scripts/06-verify.sh            # health + generation + bench
scripts/07-quality-gate.sh      # greedy math gate (17*23+45 == 436)
scripts/qwen-dflash2-rank.sh    # the actual docker run (single rank)
bench/edit_bench.py             # C1 edit-heavy/fresh benchmark (from engine counters)
bench/quality_capture.py        # 6-prompt greedy capture
bench/prod_tps.py               # simple client-side tps probe
docs/architecture.md            # how the pieces fit, RoCE env details
docs/README-agent.md            # runbook for AI agents operating this stack
docs/results.md                 # full benchmark tables + methodology

Step-by-step (with what each does)

0. Prereqs

1. Download models (~30 min, both nodes)

2. Build image (~1–2 h, node1)

3. lm_head surgery (~10 min, node1)

4. Replicate to node2 (~5 min)

• draft dir to node2.

4b. Container image to node2 — don't skip (see step 2 note).

5. Launch pair (~4–5 min to health)

6–7. Verify

Pitfalls (learned the hard way — read before debugging)

1. lm_head must be BF16 — DFlash2's selector TopKs over target logits; the image asserts UnquantizedEmbeddingMethod. Dequant via 02. (This is the headline fix; every public NVFP4 Qwen3.8 checkpoint ships a quantized lm_head.)
2. Quant-config precedence: config_groups.group_0.targets overrides the ignore list. Your ignore: ["lm_head"] does nothing while re:.*lm_head sits in targets. 02 strips it from targets AND adds it to ignore. (Cost me three failed boots to find.)
3. Hardlink bleed: shutil.copytree(copy_function=os.link) keeps the original and the copy sharing inodes — a later json.dump into config.json rewrites the shared inode and corrupts the original checkpoint. Always write-temp-then-os.replace() configs (02 does).
4. Copy path exists: 02 runs copytree into an existing dir → symlink loops. It removes .safetensors in DST first; if DST is partial, delete it and re-run. 4b. Served model name is exactly qwen3.8-27b-dflash2 (dot-separated, set by --served-model-name in the rank script). Clients must match it.
5. NCCL/RoCE env: exact working set is in qwen-dflash2-rank.sh (NCCL_IB_HCA=rocep1s0f1, RoCE v2, NCCL_CUMEM_ENABLE=0, VLLM_MARLIN_USE_ATOMIC_ADD==1 etc.). Don't hand-tune without capturing a baseline first.
6. Health timing: boot → 200 takes ~4–5 min. My first 650 s health-poll loop assumed faster; don't panic at 200 s.
7. TP2 MPI-rank pairing: rank1 must be up before rank0 starts serving (rank1 joins via --headless). 04 launches r1 first for this reason. If you restart rank0 only, expect a stuck handshake → restart both, r1 first.
8. Stale background jobs: if you automate around this stack, ensure old launcher jobs are dead before relaunching — a zombie once docker rm -f'd a healthy pair and relaunched the wrong checkpoint.
9. launchd/kickstart gotcha (Mac LiteLLM gateways): launchctl kickstart -k recycles only the worker; the master keeps the old config. Kill the master PID and let launchd respawn for config changes to take.
10. K must be multiple of 8 (draft block_size). K=16 peak on GB10.

Credit & lineage

• DFlash2: z-lab (draft weights + vLLM fork). Blog: inco.ai/blog/dflash2.
• NVFP4 target: unsloth/Qwen3.8-27B-NVFP4.
• vLLM PR 52816 by SubSir (z-lab), open as of 2026-08-19.
• This repo = integration engineering + the lm_head BF16 fix + GB10 K-sweep + 2-node RoCE recipe. MIT licensed. Benchmarks by Chris Fontes.