NVFP4 build of Ornith-1.0-35B-A3B set up for DFlash speculative decoding in vLLM. The point of this repo is not the weights — it is the recipe: how to get NVFP4 + DFlash + FP8 KV + long context + tensor-parallel all running at the same time on Blackwell hardware that has no native FP4 (GB10 / sm_121 and dual RTX 5070 Ti / sm_120).

On 2× 16 GB the KV pool measures ~347k tokens — a 35B agent model with room for up to ~256k single-stream context. Everything here is benchmarked and verified at 128k (256k is within the pool budget, not yet load-tested end to end).

That combination is fiddly. Getting each piece alone is easy; getting all four at once has exactly one thing that makes it work, and a pile of dead ends. This repo documents both.

Base model: Ornith-1.0-35B-A3B (Qwen3.6-35B-A3B family), NVFP4 (modelopt), MIT
Draft model: z-lab/Qwen3.6-35B-A3B-DFlash — attached at serve time, not re-uploaded here (Apache-2.0)
Engine: vllm/vllm-openai:nightly — the one image where this works (see below)
Sibling repo (MTP variant): pottokao/Ornith-1.0-35B-Heretic-NVFP4-MTP — same model with a trainable MTP head instead of DFlash

TL;DR — the one thing that matters

The engine image. DFlash on a no-native-FP4 Blackwell card only comes together on a vLLM nightly build:

image	result
`ghcr.io/aeon-7/aeon-vllm-ultimate`	DFlash forces FlashInfer → `Window left is not the same for all layers` on the GDN/mamba hybrid → ✗
`vllm/vllm-openai:muse-glimmer`	allows DFlash + TRITON_ATTN, but has no sm_121 kernels → CUDA device-side assert on GB10 → ✗
`vllm/vllm-openai:nightly`	has sm_121 and DFlash works with `TRITON_ATTN` → ✓

Pin the digest (nightly moves):

# verified on GB10 / sm_121
vllm/vllm-openai@sha256:5bda7078b1bb17f74d369e3ded63115a77d5ea5eeb9eab6ca9a52d108f9a262d   # vLLM 0.23.1rc1.dev968+g2c17d33f4
# verified on dual RTX 5070 Ti / sm_120 (TP2)
vllm/vllm-openai@sha256:a671d5fcda70fe9ac6f245f9780821de459fb4ee22c018fd07a0f10a55279bf9   # vLLM 0.23.1rc1.dev1000+g95ed0feaa

Quick start

GB10 / DGX Spark (sm_121, ~121 GB unified) — memory free, path is the wall

Two separate things, and only the memory is easy:

Memory: nothing to optimise. Set a high utilisation and go; pick your own context / concurrency. The memory tricks further down are a 16 GB problem GB10 doesn't have.
Path: the same image gate from the top — and sm_121 is where it bites hardest. muse-glimmer has no sm_121 kernels (CUDA assert, sm_121-specific); aeon forces FlashInfer (Window-left crash). Use the nightly sm_121 digest, and keep the backends exactly: TRITON_ATTN (main and in speculative-config), --moe-backend humming, FP8 KV. sm_121 also has no native FP4 — dense NVFP4 falls to Marlin (weight-only), expected; don't hunt for a faster FP4 path that doesn't exist.

We verified only that NVFP4 + DFlash runs clean and fits on GB10 — we did not tune it.

bash

docker run -d --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/Ornith-DFlash:/model \
  -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai:nightly \
  serve /model --trust-remote-code \
    --quantization modelopt --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
    --max-model-len 131072 --gpu-memory-utilization 0.85 \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":4,"attention_backend":"TRITON_ATTN"}'

Dual RTX 5070 Ti (sm_120, 16 GB × 2, TP2) — the tight recipe

This is where every flag earns its place. Working config, 128k context + cudagraph:

bash

docker run -d --gpus all \
  -e NCCL_P2P_DISABLE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/Ornith-DFlash:/model \
  -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai:nightly \
  serve /model --trust-remote-code --tensor-parallel-size 2 \
    --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --moe-backend humming \
    --gpu-memory-utilization 0.95 --kv-cache-memory 3200000000 --max-model-len 131072 \
    --max-num-seqs 2 --max-num-batched-tokens 2048 --enable-chunked-prefill \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":4,"moe_backend":"humming","attention_backend":"TRITON_ATTN"}'

Backends — what must be set, and why (16 GB TP2)

Each of these, if wrong, either crashes or silently halves your speed:

flag	value	why
`--attention-backend` (main)	`TRITON_ATTN`	only backend that does FP8 KV and the GDN/mamba mixed-window layout and FULL cudagraph. `flash_attn` has no FP8 KV; `flashinfer` crashes on mixed windows under spec-decode
`attention_backend` (in `speculative-config`)	`TRITON_ATTN`	the drafter does not inherit the CLI flag. Left alone it picks FlashInfer, which only supports `UNIFORM_SINGLE_TOKEN_DECODE` and drags the whole engine down to PIECEWISE cudagraph
`--moe-backend` + `moe_backend`	`humming`	NVFP4 MoE on a no-native-FP4 card. `triton` is rejected for NVFP4 MoE. (Dense NVFP4 linears still fall to Marlin weight-only — unavoidable, sm_120/sm_121 have no FP4 tensor cores)
`--kv-cache-dtype`	`fp8`	load-bearing. BF16 KV doubles per-token cost; without FP8 KV, 128k does not fit in 16 GB. FP8 KV coexists with DFlash + cudagraph — verified
`--max-num-batched-tokens`	`≤ 2048`	caps the prefill activation spike. At 8192 the activation peak alone is ~1.15 GB/card and you OOM; at 2048 it is ~0.4 GB. Chunked-prefill feeds a 128k prompt in 2048-token slices so there is never a big one-shot activation
(no) `--enforce-eager`	omitted	keep cudagraph. Eager is 38 tok/s; FULL cudagraph is 217 tok/s (5.7×). Costs ~0.28 GB/card — make room by not maxing `--kv-cache-memory`

Memory model (the part that surprised us)

On the GDN/mamba hybrid, the KV pool size is set by --kv-cache-memory, not by --max-num-seqs.

--max-num-seqs 2  →  KV pool 347,124 tokens,  free 561 MiB/card
--max-num-seqs 4  →  KV pool 347,124 tokens,  free 359 MiB/card   (same pool!)

max-num-seqs is almost free — each concurrent sequence just reserves one mamba-state slot (~100 MiB/card). So the real picture is a single shared budget you split three ways:

The ~347k-token pool → spend it on whichever you need:

context — up to ~256k single-stream (--max-model-len)

concurrency — more --max-num-seqs (nearly free, ~100 MiB/card each)

speculation depth — raise num_speculative_tokens (K) 4→6+ for longer accept length

context × concurrency ≤ pool. Want a bigger pool → raise --kv-cache-memory until free memory bottoms out. It's very flexible — tune to your workload.

The memory hog is not DFlash's draft (737 MB) — it is the target model's own GDN/mamba linear-attention state (30 of 40 layers), which is per-sequence and does not shard across TP ranks (Setting attention block size to 2160 tokens to ensure page size ≥ mamba page size).

128k / cudagraph / concurrency — pick your point (all measured, TP2 16 GB × 2)

goal	max-model-len	single-stream	AGG	KV pool	free/card
128k, cudagraph, 1 stream	131,072	217.8 tok/s	—	282k	1,149 MiB
128k, cudagraph, 2 streams	131,072	220.9	332.1	347k	561 MiB
128k, eager (don't)	131,072	38 ✗	—	325k	—

The pool holds ~347k tokens, i.e. roughly 256k of usable single-stream context if you want it. What to do with the headroom over 128k is your call — spend it on longer context (up to ~256k) or on more speculation (num_speculative_tokens 4 → 6+). z-lab suggests block size 8 (throughput) or 16 (single-user accept length); 4 here is deliberately conservative.

Note: with speculative decoding, higher concurrency does not always mean more throughput — rejected drafts burn verify compute, so AGG can fall as you add streams. Measure your own load.

Benchmarks — DFlash vs MTP, both platforms

Same harness both platforms: vllm bench serve (Spec-Bench), concurrency 1, 20 warmups, --ignore-eos, enable_thinking:false, K=4. mine/* = held-out agent-code (the judges); spec/* = Spec-Bench guard categories (regression check). MTP baseline = the sibling MTP repo.

tok/s is a property of the hardware; acceptance length (acc) is a property of the model. Read both.

Dual RTX 5070 Ti (sm_120, TP2, 16 GB × 2)

category	DFlash tok/s	DF acc	MTP tok/s	MTP acc
mine/code	133.89	2.69	132.69	2.86
mine/agent	165.98	3.40	162.35	3.43
spec/summarization	162.80	3.04	162.62	3.06
spec/rag	178.72	3.47	178.85	3.53
spec/coding	272.08	3.51	274.61	3.55
spec/translation	252.17	3.27	266.54	3.34
spec/qa	234.87	2.93	243.82	3.03
spec/math_reasoning	279.01	4.03	300.51	3.86

Dead even on speed (DFlash marginally ahead on the two agent judges); acceptance a hair lower for DFlash. DFlash's real edge here is the KV pool + 128k, which MTP can't match on 16 GB.

GB10 / DGX Spark (sm_121, single card)

category	DFlash tok/s	DF acc	MTP tok/s	MTP acc
mine/code	60.88	2.69	65.44	2.86
mine/agent	73.54	3.17	75.87	3.30
spec/summarization	70.52	3.03	74.28	3.16
spec/rag	80.26	3.57	83.17	3.58
spec/coding	93.41	3.60	89.97	3.53
spec/translation	79.33	3.29	79.82	3.26
spec/qa	76.73	2.89	80.61	3.02
spec/math_reasoning	92.24	3.99	91.17	3.91

On a single card MTP is marginally ahead on most categories (DFlash ~93–98% of MTP tok/s, acc a touch lower), except math/coding where DFlash edges up. Essentially tied — which matches the folk wisdom: DFlash ≈ MTP on speed, slightly behind on acceptance, and it buys you the context.

DFlash vs MTP — when to use which

Not a knockout — a fit question.

	MTP	DFlash
out of the box	needs a matched/aligned head	attach and run (z-lab's is pre-trained)
ceiling	trainable → push acceptance on your own distribution	fixed draft, retraining is involved
KV / context	carries a head → smaller pool → shorter context	headless → bigger pool → 128k (up to ~256k)
best for	fixed distribution, willing to train, context is enough	agent / long-context, no training

Verdict on 16 GB TP2: DFlash ties MTP on speed (slightly ahead on the agent-code judges), is marginally lower on acceptance, and gives a much larger KV pool + 128k. Since KV is gold for agents (long tool histories, big system prompts), DFlash is the better no-training default here. MTP still wins if you invest in training and don't need the context.

Compatible, but unverified

The recipe is a serve-time attachment for the Qwen3.6-35B-A3B family — any model on that base can load the same z-lab draft (sakamakismile/Qwen-AgentWorld-35B-A3B, huihui variants, …).

Qwen-AgentWorld-35B-A3B is a natural target (headless, agent-trained, 262k context window). But treat these as variables to test, not claims:

The z-lab draft was trained against the base distribution; on an agent-finetuned model the acceptance rate may drop — measure it.
AgentWorld ships as compressed-tensors W4A4 (not modelopt) and is multimodal (vision tower in full precision) → the flags and the memory budget differ. Untested.

Credits & licenses

Ornith-1.0-35B-A3B — base model (MIT)
z-lab/Qwen3.6-35B-A3B-DFlash — the draft model that does all the speculation (Apache-2.0). All credit for DFlash to z-lab.
vLLM — engine (Apache-2.0); the nightly build is what makes this possible.

MIT (base) + Apache-2.0 (draft) — compatible. This repo adds no new model; it packages a base and a run recipe.

pottokao/Ornith-1.0-35B-Heretic-NVFP4-DFlash

Ornith-1.0-35B-Heretic-NVFP4-DFlash