Qwen3.5-0.8B DSpark drafter (v0.3, experimental)

Card status (2026-07-25, corrected): all numbers below are measured. The matched-vs-generic A/B has landed (see its section) and the vLLM smoke test passed (see How to run). 2026-07-31: vLLM acceptance fixed and verified at llama.cpp parity (0.155 vs 0.158 per drafted token). The earlier vLLM shortfall was a config.json rope bug in this repo (partial_rotary_factor inherited from the target), not a runtime difference; the shipped config now sets it to 1.0 explicitly. See the vLLM and cross-stack sections. Correction: an earlier version of this card described a "quant tax" and claimed a bf16-trained head loses most of its acceptance on aggressive quants. Follow-up work refuted that: acceptance tracks whether the target is still coherent, not the precision gap between head and target. The affected sections are rewritten below, with the evidence.

v0.3: the supervision-volume line is now three points, still rising

• Monotone in every cell, v0.1 to v0.2 to v0.3. P1 acceptance is +50% relative over v0.2.
• Output identity across heads: under greedy decoding all three heads produce the same target-verified completion per (target, prompt); the output hash is identical across v0.1/v0.2/v0.3. Speculative decoding is lossless when the target verifies. The head changes how much draft is accepted, never the text.
• No plateau at 28k. The next volume step remains the cheapest lever before architecture changes:

Drafter-side quantization is verification-lossless, now shown at 744B scale

v0.2: supervision volume, measured (with a corrected attribution)

Correction (2026-07-28): the first version of this section claimed v0.1 trained on generic dataset responses and v0.2 on target-regenerated ones, framing this as a distribution-matching experiment. That was wrong: both versions trained on self-distilled data (responses regenerated by the target itself, same public prompt pool, same temperature). The real difference, found by auditing the training files: v0.1's regeneration capped responses at ~110 tokens on average (426 chars), while v0.2's ran to full length (~3,500 chars avg). v0.2 saw ~9x the supervised tokens, almost entirely from response length. The acceptance gains are real; the correct attribution is supervision volume (and long-form responses), not distribution matching.

• The v0.2 head also drafts fewer tokens while landing more (626 drafted vs 759 on the code probe): it wastes fewer draft passes, which is where the throughput gain comes from.
• Integrity anchor: in the same session, v0.1 reproduced its published CUDA reference byte-exactly (0.11594, 88/759: cross-backend greedy determinism), so these deltas sit on a zero-noise floor.
• Scope honestly: absolute acceptance stays low (0.17-0.20) because a 0.8B target is the hardest case for speculation (see the loop-rate and economics sections). This experiment proves the method at constant cost; the same recipe on larger targets is where the absolute numbers get interesting.
• Mechanism note, post-correction: this result is about how much supervised signal the drafter sees per sample (full-length responses vs truncated ones), on top of already-matched self-distilled data. Related but distinct mechanisms measured elsewhere in this project: own-data imatrix calibration helps target quants (27B card), and real importance data made no difference to MTP-block quantization at q2_K on a 27B. Measure each axis; do not assume one story covers them all.
• Control caveat, stated rather than buried: the two runs differ in one secondary knob. A config drift halved the global batch between runs (v0.1: 256, 450 optimizer steps; v0.2: 128, 1,020 steps), so v0.2 took more, smaller optimizer steps over the same 10 epochs. The data change is the dominant variable, but the batch change is a real confound this experiment did not isolate; v0.3 pinned the batch config (global batch 128, same as v0.2) and the gain held.
• The training data is published: satgeze/Qwen3.5-0.8B-selfdistill carries both versions' pairs with full provenance (prompts: open-perfectblend, Apache-2.0; responses: generated by the target, unique to that dataset).
• Training data: 13,273 samples regenerated via llama-server (bf16, 16 parallel slots) using the fork's generate_train_data.py --resume; features captured with the GGUF capture pipeline documented in the training guide linked below.

How to run it

llama.cpp (tested, all numbers on this card were measured with this)

vLLM (verified: full acceptance)

SGLang

Measured results (llama.cpp, RTX Pro 6000 Blackwell, greedy, 200-token completions)

Apple Silicon (M3 Max 128GB, Metal, same GGUFs, same probes)

Economics across the full target-quant ladder (measured, M3 Max Metal)

SPEED-Bench (qualitative set, coding+writing, temperature 0, concurrency 1, M3 Max Metal, bf16 target)

Loop rate: this target degenerates on long greedy generation, at every precision

Target quantization ladder, measured on this head

1. At aggressive quantization, acceptance falls when the target itself degrades. This is a small-model effect: a 0.8B at 2.6 bpw and a 4B at 1.6 bpw are broken models, and no drafter can predict a broken model. A 27B at 1.6 bpw with a calibrated imatrix keeps Q8-level acceptance using this same bf16-trained-head recipe. Before blaming precision, check whether the target is still coherent (perplexity and loop rate both help).
2. The fix is to self-distill and train the head against the exact deployed artifact (the quantized GGUF actually served), not the full-precision base. Our llama.cpp capture tool + DeepSpec-format cache builder exist for exactly this (validated at cosine >= 0.9998 against the HF capture pipeline; full how-to guide); matched-vs-generic A/B numbers land here as they are measured. The same applies to modified variants (e.g. abliterated models): any distribution shift in the target taxes a mismatched head.
3. When evaluating published heads, compare targets at the precision the head was trained against before concluding anything about the head or the runtime.

Matched-vs-generic A/B: does capture precision matter? (measured)

1. Pipeline validation: the bf16-GGUF-captured head reproduces the original HF-cache-trained v0.1 head (0.126/0.152 vs 0.116/0.152) within probe noise, and their final training losses are statistically identical (1.646 vs 1.63). Training from llama.cpp-captured features is a full substitute for the HF capture pipeline, proven through training, not just feature cosine.
2. No matched-head advantage at Q8: all four cells sit within probe variance. This is the expected result: Q8 features drift from bf16 by cosine ~0.999, so there is nothing for a matched head to learn. The 27B ladder later closed this out further: at 1.6 bpw with a calibrated imatrix, the same bf16-trained recipe still holds Q8-level acceptance, so the "matched training pays off at low bit-width" half of this remains a hypothesis we have not confirmed. Published matched-vs-unmatched A/Bs elsewhere measure 0-2% (arXiv 2607.04244 Table 4: 4.92 vs 4.97 accepted length; SpecForge PR #318: 7.10 vs 7.11).

Cross-stack controlled A/B: the same head in llama.cpp and vLLM (measured)

• Lesson carried forward: acceptance numbers do not transfer between stacks until the drafter's config semantics are verified against how the head was actually trained. Measure your own stack pair before committing to serving numbers.
• Throughput is deliberately not compared here. At 0.8B, speculation is a net slowdown in wall-clock terms regardless of stack (see the economics section); this A/B isolates the acceptance mechanics only.
• Reproduction note: in llama.cpp, --spec-type draft-dspark is required. Without it the server loads the draft model and then silently serves without speculation; the only visible differences are a missing draft acceptance line in the per-request timings and faster-than-expected raw decode.

Official DeepSpec eval (per-proposal acceptance, temperature 0, step 450)

How this head was trained

1. From-scratch heads need far more than 12K samples. The community curve (each point = that head on its own target): 2.4K samples = 1.08 accepted length (failed), 10K warm-started = 1.51, 250K = 3.34, ~1M = 4.6-4.8. v0.1 lands where the curve predicts for cold-start 12K. Warm-starting is the highest-leverage fix (our 27B head proves it at the same data scale).
2. Small targets are poor speculation hosts on fast hardware (see the economics ladder above). Small-target heads make sense on slow/edge hosts, not on datacenter GPUs.

Recipe (reproducible)

• Data: 12,000 prompts from mlabonne/open-perfectblend, responses regenerated by the target itself (llama-server bf16 GGUF, temperature 0.7, top-p 0.8, top-k 20, thinking disabled, max 2048 tokens).
• Cache: DeepSpec prepare_target_cache (capture layers [1, 6, 11, 16, 21] of 24), 11,577/11,722 valid.
• Training: DeepSpec Qwen3DSparkTrainer, block_size 7, 5 draft layers, markov_rank 256, lr 6e-4, global batch 128, 5 epochs (450 steps), bf16, single GPU. Final loss 1.63 (from 2.86).
• Conversion: llama.cpp branch satindergrewal/llama.cpp:dspark-qwen35 (adds qwen3.5-family DSpark target+drafter support on top of PR #25173), then served with --spec-type draft-dspark --spec-draft-n-max 7.

Credits