You can deploy this model and start earning money today!

MirecX/
Qwen3.5-397B-A17B-L3H5-DFlash

Name: Qwen3.5-397B-A17B-L3H5-DFlash
Rating: 2.5 (11 reviews)

safetensors

qwen3

custom_code

Views

No views yet

Qwen3.5-397B-A17B-L3H5-DFlash

Experimental DFlash speculative-decoding drafter for QuantTrio/Qwen3.5-397B-A17B-AWQ.

⚠️ Highly experimental — for tinkering, not production-grade. Trained on a small on-policy corpus (~226 shards = math/stem prompts from Nemotron-Post-Training-V2 + real coding-agent session prompts). Acceptance is well below paper-level due to the small training pool, but eval and runtime numbers should now match (no longer overstating).

Shape

3 drafter layers, hidden_size=4096, ~0.65B params, bf16
target taps: layers [2, 16, 30, 43, 57] of the 60-layer target (paper pattern)
block_size=16
all full_attention (target has SWA but drafter doesn't use it at this scale)

Eval (50 held-out shards = 30 nemotron + 20 coding-agent, 200 blocks, greedy verification)

mean accept length: 1.94
per-position-1 (k=1) hit rate: 70.5%
cumulative top-1 (% of blocks accepting at least k tokens):

k 1 2 3 4 5 6 8 10 12 15
% 70.5 44.5 27.5 15.0 8.5 6.0 3.5 2.5 2.0 1.5

The drafter is strongest on templated outputs (LaTeX math, markdown lists, common phrasing — full 15-token chains seen on those) and weaker on free-form code/chat continuations. Mid-block acceptance is real (k=4 at 15%), so on inference you'll see runs of 4–8 tokens regularly with occasional full-block 15+ accepts.

Use with vLLM

--speculative-config '{"method":"dflash","model":"MirecX/Qwen3.5-397B-A17B-L3H5-DFlash","num_speculative_tokens":6}'

num_speculative_tokens of 4–8 work well; runtime acceptance plateaus past about 8 with this drafter.

Training

corpus: 166 nemotron (math/stem/chat/code mix) + 60 coding-agent session prompts, all on-policy from the QuantTrio AWQ target
10000 steps, bs=1, grad-accum=2, anchors-per-seq=6, max-ctx=uncapped, loss-decay=0.85, warmup=500, ckpt-every=2000
best ckpt: step_10000
training: RTX 5080 16 GB, ~14 min

Capture path

This drafter was trained on hidden states captured via the patched dflash_extract plugin, which captures the full residual stream (hidden_states + residual at each tracked layer) — matching what vLLM's spec-decode reads at runtime. Earlier versions of this repo were trained on hidden_states only and exhibited a 3× eval-vs-runtime acceptance cliff; that's been fixed.

Companion: MirecX/Qwen3.5-397B-A17B-L5H5-DFlash (5-layer drafter; underperforms L3H5 at the current data scale, included for reference).