Views
No views yet
ImportantThis repo's checkpoints are v11 (default branch /main). Older v10 checkpoints remain on thev10branch. Use the matching code release: v11 checkpoints requirespeculative_pipeline_decodingv11; v10 checkpoints load viaold_version_v10/.
.pt file is a single checkpoint produced by training; pair it with the same base model architecture it was trained on (see config["base_model_path"] inside the file).{model}_s{num_stages}_l{num_spec_layers}.pt| Part | Meaning |
|---|---|
{model} | Base model tag from training config (e.g. Qwen3.5-4B, Qwen3.5-9B) |
s{...} | num_stages — pipeline depth (number of target-model stages) |
l{...} | num_spec_layers — number of Transformer layers in the speculation module |
Qwen3.5-9B_s16_l2.pt → Qwen3.5-9B base, 16 stages, 2 spec layers.1{
2 "state_dict": ..., # weights of the speculation module
3 "config": { ... }, # hyperparameters and metadata
4}config fields (always present)| Field | Description |
|---|---|
base_model_path | Base model path recorded at training time (often a machine-local path; override at load time — see below) |
hidden_size | Hidden size (matches base model) |
vocab_size | Base model vocabulary size |
draft_vocab_size | Draft head output size (full vocab or draft subset) |
num_stages | Pipeline depth (same as s in filename) |
num_spec_layers | Speculation module depth (same as l in filename) |
version | Checkpoint format version (11) |
num_aggr_types | Number of aggregation types m in the speculation module, determining the number of FC modules |
aggr_feature_bound | HF hidden-state layer indices for aggregation anchors g_0..g_{m-1} (replaces v10's shallow_hidden_layer_indices) |
trained_with_use_deepest | Whether training used deepest-layer features |
config fields (optional)| Field | Description |
|---|---|
model_type | Base model type recorded at training time (e.g. qwen3_5) |
spec_init_from_base_layers | Base layers used to initialize the spec module (if any) |
draft_token_ids | Draft vocabulary token ids (only when trained with a draft vocab subset) |
| Base model | s (stages) | l (spec layers) | Filename |
|---|---|---|---|
| Qwen3.5-4B | 4 | 4 | Qwen3.5-4B_s4_l4.pt |
| Qwen3.5-4B | 8 | 4 | Qwen3.5-4B_s8_l4.pt |
| Qwen3.5-4B | 16 | 2 | Qwen3.5-4B_s16_l2.pt |
| Qwen3.5-9B | 4 | 4 | Qwen3.5-9B_s4_l4.pt |
| Qwen3.5-9B | 8 | 4 | Qwen3.5-9B_s8_l4.pt |
| Qwen3.5-9B | 16 | 2 | Qwen3.5-9B_s16_l2.pt |
config["base_model_path"] is often a local path from the training machine (e.g. /share/models/Qwen3.5-4B). On your machine, pass the correct Hugging Face id or local directory via --base_model_path; it overrides the path stored in the checkpoint:1python pipeline_inference.py \
2 --spec_head_ckpt /path/to/Qwen3.5-4B_s4_l4.pt \
3 --base_model_path Qwen/Qwen3.5-4B
4
5python eval.py \
6 --spec_head_ckpt /path/to/Qwen3.5-4B_s4_l4.pt \
7 --base_model_path /your/local/Qwen3.5-4B \
8 --data_dir eval_data \
9 --output_dir ./eval_output--base_model_path is omitted, the value from config["base_model_path"] is used as-is.1@misc{yu2026speculativepipelinedecodinghigheraccruacy,
2 title={Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism},
3 author={Yijiong Yu and Huazheng Wang and Shuai Yuan and Ruilong Ren and Ji Pei},
4 year={2026},
5 eprint={2605.30852},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2605.30852},
9}