Views
No views yet
train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by Qwen3-8B (with reasoning). Training compute for this model was sponsored by Modal.1# In virtual environment with speculators installed
2python scripts/prepare_data.py \
3 --model Qwen/Qwen3-8B
4 --data ./regenerated_data.jsonl \
5 --output ./output \
6 --seq-length 81921# In (separate) virutal environment with vllm installed
2CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
3 Qwen/Qwen3-8B \
4 --target-layer-ids 2 10 18 26 34 \
5 -- --port 8000 \
6 --gpu-memory-utilization 0.9 \
7 --disable-uvicorn-access-log \
8 --tensor-parallel-size 1 \
9 --data-parallel-size 21# In virtual environment with speculators installed
2CUDA_VISIBLE_DEVICES=2,3 torchrun \
3 --standalone \
4 --nproc_per_node 2 \
5 scripts/train.py \
6 --verifier-name-or-path Qwen/Qwen3-8B \
7 --speculator-type dflash \
8 --num-layers 5 \
9 --data-path ./output \
10 --vllm-endpoint http://localhost:8000/v1 \
11 --save-path ./output/checkpoints \
12 --epochs 3 \
13 --lr 0.0006 \
14 --total-seq-len 8192 \
15 --on-missing generate \
16 --on-generate delete \
17 --seed 42 \
18 --log-freq 100 \
19 --draft-vocab-size 32000 \
20 --draft-arch qwen3 \
21 --target-layer-ids 2 10 18 26 34 \
22 --draft-hidden-act silu \
23 --scheduler-type cosine \
24 --max-anchors 3072 \
25 --prefetch-factor 2 \
26 --num-workers 8| Base Model | Qwen/Qwen3-8B |
| Chat Template | Qwen/Qwen3-8B (use /chat/completions endpoint) |
| Format | Safetensors |
| License | Apache 2.0 |
| Validation Hardware | Nvidia H100 |
1# Install vLLM from the required PR
2pip install git+https://github.com/vllm-project/vllm.git@refs/pull/41880/head
3
4# Deploy with speculative decoding
5vllm serve Qwen/Qwen3-8B \
6 --tensor-parallel-size 1 \
7 --max-model-len 16384 \
8 --speculative-config '{
9 "model": "RedHatAI/Qwen3-8B-speculator.dflash",
10 "num_speculative_tokens": 7,
11 "method": "dflash"
12 }'| Dataset | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Avg Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 79.9% | 58.0% | 40.3% | 27.0% | 17.8% | 11.3% | 6.8% | 3.410 |
| math_reasoning | 82.2% | 62.7% | 46.2% | 33.5% | 23.4% | 15.8% | 9.9% | 3.740 |
| qa | 68.9% | 42.6% | 25.0% | 14.4% | 8.1% | 4.4% | 2.3% | 2.660 |
| question | 73.0% | 47.6% | 30.1% | 18.9% | 11.7% | 7.1% | 4.1% | 2.930 |
| rag | 71.1% | 44.8% | 27.0% | 15.7% | 8.9% | 4.9% | 2.5% | 2.750 |
| summarization | 65.5% | 36.1% | 19.0% | 9.5% | 4.7% | 2.3% | 1.1% | 2.380 |
| tool_call | 71.3% | 44.6% | 25.8% | 14.4% | 7.8% | 4.1% | 2.1% | 2.700 |
| translation | 63.8% | 38.4% | 22.1% | 11.8% | 6.1% | 3.2% | 1.5% | 2.470 |
| writing | 73.2% | 47.7% | 30.1% | 18.9% | 11.8% | 7.2% | 4.2% | 2.930 |