GLM-5.2 DSpark speculator — finetuned for QuantTrio Int4-Int8Mix
A finetune of
RedHatAI/GLM-5.2-speculator.dspark
re-aligned to the hidden states of the
int4/int8 target
QuantTrio/GLM-5.2-Int4-Int8Mix.
The public speculator was trained against FP8-GLM hidden states; served
against the QuantTrio quant it accepts ~1.45 extra tokens/step. This finetune
reaches ~2.1, which on a 4× DGX Spark (GB10) cluster is the difference
between ~19.6 and ~24.7 tok/s decode at up to 1,048,576 context —
lossless (exact rejection sampling), acceptance flat in depth.
Full serving recipe:
bird/GLM-spark → recipes/4x-dspark-1m
(vLLM branch
bird/vllm-lil@dspark-ring-1m-20260711
with ring-buffer windowed drafting — required; stock vLLM's full-context
DSpark port suffers acceptance depth-collapse).
Files
| path | what |
|---|
model.safetensors + config.json | full-vocab finetune — exact drop-in for the RedHatAI original (same speculators layout) |
ft32k/ | reduced draft vocab (top-32,768 ids, 99.94% coverage, d2t mapping) — what our production runs; same speed, ~400 MB/rank less memory |
Usage (vLLM, with the recipe branch)
1--speculative-config '{"model":"/path/to/this-repo","num_speculative_tokens":3,
2 "attention_backend":"TRITON_ATTN","quantization":"fp8"}'
3# plus: VLLM_DSPARK_DRAFT_RING=1 VLLM_DSPARK_DRAFT_WINDOW=1024
num_speculative_tokens must keep 1+k a power of two on the b12x path
(k∈{3,7}); k=3 is optimal.
Training
Teacher-forced on ~3M tokens of self-generated capture from the target quant
itself (raw per-step target aux hidden states + token ids recorded while
serving natural technical text at depths up to ~880k tokens), using the
pipeline published in the recipe (speculator-training/): stitched
full-window streams, bonus-anchor fill-in readout, window-excludes-anchor,
length-weighted depth sampling, best-rolling-checkpoint selection. 2,000
steps, 3× GB10 DDP, ~75 minutes. Numerically verified against the inference
kernels (bit-exact hidden-norm, bf16-noise rope at 1M positions).
Measured (2026-07-11, 4× GB10, TP4+DCP4, 1M pool)
- Decode: 24.4/24.4/24.6/24.3/25.2/24.8 and 25.6/24.1/24.5 tok/s across two
boots (greedy 2k-token completions) — mean 24.66
- Acceptance: 2.03–2.23 extra tokens/step, flat 16k→486k at fixed content
- vs base speculator on the same stack: accept ~1.45, ~19.6 tok/s
Quality is unaffected by construction: DSpark verification is exact
rejection sampling against the target.
