Qwen3.6-35B-A3B-NVFP4 · DSPARK draft model (v2)

v2 — a later training checkpoint than v1, +6.8% acceptance across the board (Spec-Bench macro 3.70 → 3.96, every domain improved). v1 remains available at the link above for anyone pinned to it. Training continues; a final version will follow.

Overview

⚠️ Serve the frozen target Koopah/Qwen3.6-35B-A3B-NVFP4, not upstream main. On 2026-07-10 the upstream repo silently replaced its weights with a mixed-precision re-export that produces degenerate looping output in our testing (two SM120 GPUs, all KV dtypes) and is NOT the checkpoint this draft was trained against. The frozen repo is byte-identical to upstream revision 612d523c58 — the artifact every number on this card was validated with.

🚧 Training is still in progress. This is the best-scoring checkpoint from an ongoing multi-week run, published early. Acceptance is still climbing — expect updated weights.

Performance at a glance (single GPU)

sglang (fork branch below)

vLLM 0.26

llama.cpp (single-user / local serving)

Verification (SPEED-Bench, llama.cpp)

Continuous serving (closed loop)

Evaluation detail

Spec-Bench (6 domains, 480 prompts)

The -Fast variant

Deployment (sglang)

• Patched fork branch: KoopahTManiac/sglang@dspark-hybrid-nvfp4 — last known-good upstream commit for NVFP4-on-Blackwell, plus four commits on top: the hybrid-GDN state-commit fix for DSPARK, a backport of upstream's DSPARK grammar-constrained decoding (#31753) — so JSON-schema / regex / EBNF output and tool_choice work through speculative decoding — and a backport of the NEXTN mamba-track guard (#27998), so the built-in MTP head also serves on this branch for A/B comparisons. Diff it yourself.
• A ready-to-build Dockerfile in this repo: a thin overlay on the official lmsysorg/sglang:nightly-dev-20260714-2f79d334 image (the last engine era before the SM120 NVFP4 regression, with era-matched kernels and the flashinfer JIT cache prebaked) that swaps in the fork branch's Python tree — all fork commits are pure Python. The default command carries the full tuned flag set and env; validated on an RTX PRO 6000 Blackwell: containerized serving matches native throughput within 1% (after first-minute JIT warmup).

Deployment (vLLM)

• vLLM ≥ 0.26.0 — older versions fail to load this draft (KV page-size unification and draft-config handling both fixed in 0.26).
• --mamba-ssm-cache-dtype bfloat16 matters: without it the GDN recurrent states default to fp32 and DSpark's deep verify staging caps admission at ~47 concurrent requests (high-batch aggregate drops ~25%).
• Use the default KV-cache dtype (fp8 KV crashes on this hybrid target, see known issues).

Deployment (llama.cpp)

⚠️ Known issues in upstream engines (as of August 2026) — why the forks exist

1. Hybrid-GDN state commit (affects this target!): upstream sglang's DSPARK worker does not yet commit the target's gated-DeltaNet state after speculative verification. On hybrid-linear-attention targets like Qwen3.6-35B-A3B this silently collapses acceptance and corrupts outputs. The fix is already included in the fork branch and Docker image above (a port of DFlash's _update_target_mamba_state_after_verify into the DSPARK worker); an upstream PR will follow. Dense targets are unaffected.
2. NVFP4 on SM120 (consumer Blackwell): sglang's FP4 quantization refactor (9756f768a) routes nvfp4-pack-quantized compressed-tensors MoE onto a FlashInfer kernel path with no working SM12x support — garbage output at any temperature. Upstream work is in flight (sglang #32119, flashinfer #3932); until it lands, use the pinned fork branch above (0c01971ee base, the last known-good).
3. vLLM: fp8 KV cache crashes on this hybrid target — use the default KV dtype.
4. Confidence head unused in the vLLM/sglang numbers here: this draft ships a trained confidence head for dynamic block truncation (measured on-policy calibration: ECE 0.010, AUC 0.925). vLLM explicitly skips its weights ("not wired into inference yet"). At the time these benchmarks ran, sglang's confidence-scheduling supported MLA-attention targets only, so all vLLM/sglang numbers on this card verify the full block every round; sglang upstream has since (Aug 2026) generalized its confidence-fed ragged-verify scheduler beyond MLA — including hybrid GQA+mamba targets like this one — driven by offline-profiled SPS cost tables and STS calibration (--speculative-dspark-sps-table-path / --speculative-dspark-confidence-sts-path). Untested with this draft so far, and the path has open upstream issues (e.g. runtime- gamma and compact-graph geometry bugs), so treat it as promising but unverified. llama.cpp wires it: the llama.cpp deployment implements the DSpark paper's confidence scheduler using this head (--spec-sched, self-calibrating) — see the GGUF repo for deployment and usage guidance. vLLM/sglang integration of the head remains the open upside for mid-batch throughput vs MTP.

Model Specifications

Training Details

• Data: ~1.34 M prompts (open-perfectblend mix) with responses regenerated by the target itself in both thinking and non-thinking modes, sampled at the target's serving configuration — the draft learns the target's actual output distribution rather than generic text.
• Stack: DeepSpec, adapted for online single-GPU training against the live NVFP4 target.
• Initialization: warm-started from the z-lab DFlash backbone for this target, converted to DSPARK format (Markov head soft-started, confidence head fresh), then trained from step one under the DSPARK objective.
• Evaluation during training: rollout-based accept-length eval on held-out prompts every 100 steps at the serving sampling configuration; this repo tracks the best-scoring checkpoint.

References

• DSpark: block-diffusion speculative decoding (DeepSeek, 2026) — merged in sglang and vLLM
• DeepSpec training framework
• z-lab DFlash (backbone initialization)

Run your own benchmark