GLM-5.2 DSpark (block16) Speculator Model Card
Overview
This is a DSpark speculator model for the zai-org/GLM-5.2-FP8 base model, enabling
faster inference through speculative decoding. The architecture extends DFlash with
dual lightweight components: a Markov logit-bias head addressing token dependencies
and a per-position confidence head for acceptance-rate forecasting. Development
utilized the speculators library.
This is the block_size=16 variant — identical to
RedHatAI/GLM-5.2-speculator.dspark
except for the draft block size (16 instead of 8). See
Difference from the reference.
Model Specifications
- Base Model: zai-org/GLM-5.2-FP8
- Chat Template: GLM-5.2 (compatible with
/chat/completions)
- Format: Safetensors
- License: MIT
- Draft: 5 layers,
block_size=16, full attention, full vocabulary (154,880), aux layers [8, 23, 39, 55, 70]
- Training/Validation Hardware: 60× NVIDIA GB300 on a GB300 NVL72 rack (9 vLLM producer nodes + 6 FSDP trainer nodes, DP=24)
Difference from the dspark reference
This variant changes one knob versus RedHatAI/GLM-5.2-speculator.dspark:
block_size = 16 (versus 8).
Everything else matches: full attention, full vocabulary, aux layers
[8, 23, 39, 55, 70], lr 6e-4 cosine, 3 epochs, --max-anchors 1024, loss
{"ce": 0.1, "tv": 0.9}, Markov + confidence heads.
Checkpoint series
This repository distributes per-epoch checkpoints from a single 3-epoch training run.
The main branch tracks the most recent epoch; each epoch remains permanently
accessible as a separate revision.
| revision | epoch | status |
|---|
epoch-1 | 1 / 3 | ✅ this checkpoint |
epoch-2 | 2 / 3 | training |
epoch-3 | 3 / 3 | training |
Training Details
Training employed the Speculators library on prompts from Magpie-Align and
HuggingFaceH4 collections, with GLM-5.2-FP8 generating responses (available as
mgoin/GLM-5.2-FP8-magpie-ultrachat).
The approach is "online": the draft receives hidden states from a live GLM-5.2-FP8
vLLM server, while the trainer runs FSDP data-parallel on separate hardware. Three
commands reproduce this process — install speculators and vLLM from main branches
first.
Data Preparation
1python scripts/prepare_data.py \
2 --model zai-org/GLM-5.2-FP8 \
3 --trust-remote-code \
4 --data ./regenerated_data.jsonl \
5 --output ./output \
6 --seq-length 8192 \
7 --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'
The --assistant-pattern flag addresses GLM-5.2's inline-reasoning format where
reasoning traces remain within assistant turns; future versions may auto-detect this.
vLLM Server Launch (hidden-states server)
1CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
2 zai-org/GLM-5.2-FP8 \
3 --target-layer-ids 8 23 39 55 70 \
4 -- --port 8000 \
5 --tensor-parallel-size 4 \
6 --gpu-memory-utilization 0.9 \
7 --max-model-len 8192 \
8 --trust-remote-code
Training Command
1CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
2 --standalone \
3 --nproc_per_node 4 \
4 scripts/train.py \
5 --verifier-name-or-path zai-org/GLM-5.2-FP8 \
6 --speculator-type dspark \
7 --num-layers 5 \
8 --block-size 16 \
9 --data-path ./output \
10 --vllm-endpoint http://localhost:8000/v1 \
11 --save-path ./output/checkpoints \
12 --epochs 3 \
13 --lr 0.0006 \
14 --scheduler-type cosine \
15 --total-seq-len 4096 \
16 --draft-arch qwen3 \
17 --draft-hidden-act silu \
18 --target-layer-ids 8 23 39 55 70 \
19 --max-anchors 1024 \
20 --markov-rank 256 \
21 --enable-confidence-head \
22 --confidence-head-with-markov \
23 --loss-fn '{"ce": 0.1, "tv": 0.9}' \
24 --confidence-head-alpha 1.0 \
25 --checkpoint-freq 0.2 \
26 --on-missing generate \
27 --on-generate delete \
28 --seed 42 \
29 --log-freq 100 \
30 --prefetch-factor 2 \
31 --num-workers 8 \
32 --trust-remote-code
Notes:
- The published checkpoint was trained at scale on 60× GB300 (9 vLLM producer nodes
serving the FP8 verifier + a 6-node, DP=24 FSDP trainer) with hidden states streamed
over a Mooncake RDMA store; the command above is the equivalent single-node recipe.
- The only difference from the reference
dspark is --block-size 16 (versus 8).
- Excluding
--draft-vocab-size trains on full vocabulary; pass --draft-vocab-size 32000
to reduce it.
- DSpark-specific parameters:
--markov-rank, --enable-confidence-head,
--confidence-head-with-markov, --confidence-head-alpha. Removing these (with
--speculator-type dflash) produces standard DFlash.
- Sub-epoch checkpoints enable resumability with
--checkpoint-freq 0.2.
Deployment
vLLM nightly provides DSpark inference support:
1vllm serve zai-org/GLM-5.2-FP8 \
2 --tensor-parallel-size 4 \
3 --max-model-len 16384 \
4 --trust-remote-code \
5 --speculative-config '{
6 "model": "mgoin/GLM-5.2-speculator.dspark-block16",
7 "num_speculative_tokens": 7,
8 "method": "dspark",
9 "draft_sample_method": "probabilistic"
10 }'
Evaluation Results
End-of-epoch-1 metrics (teacher-forced validation over the held-out split):
| metric | value |
|---|
| mean accepted length | 3.741 |
| full accuracy | 0.444 |
| mean acceptance rate | 0.399 |
| confidence abs error | 0.008 |
Per-position accuracy across positions 1-7:
0.796 / 0.675 / 0.589 / 0.528 / 0.482 / 0.446 / 0.417
Acceptance length in vLLM (revision epoch-1)
[ Eval pending — to be added. ]
End-to-end measurement in vLLM speculative decoding serving GLM-5.2-FP8 with this
speculator, versus the reference dspark under identical conditions:
num_speculative_tokens=7, greedy sampling, batch size 1, 16 single-turn prompts
(8 HumanEval + 8 math reasoning), 1024 output tokens each.
| Checkpoint | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Accept Len | Decode tok/s |
|---|
epoch-1 (this repo, block16) | — | — | — | — | — | — | — | — | — |
dspark (reference, block8) | 75.8% | 55.2% | 40.3% | 29.3% | 21.5% | 15.5% | 11.1% | 3.49 | 219 |
References
- DFlash: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the parallel backbone foundation.
- DSpark (DeepSeek) — the Markov and confidence-head extensions replicated here.
- speculators — the training framework.
RedHatAI/GLM-5.2-speculator.dspark — the block_size=8 reference.
AI support contributed to pipeline development and experimental execution.
