Views
No views yet
train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by RedHatAI/Qwen3.5-397B-A17B and stored at Qwen3.5-397B-A17B-responses1# In virtual environment with speculators installed
2python scripts/prepare_data.py \
3 --model Qwen/Qwen3.5-397B-A17B
4 --data ./regenerated_data.jsonl \
5 --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
6 --output ./output \
7 --seq-length 163841# In (separate) virtual environment with [vLLM](https://github.com/imargulis/vllm/tree/fix/extract-hidden-states-hybrid-block-size) installed
2CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
3 Qwen/Qwen3.5-397B-A17B \
4 --target-layer-ids 3 15 23 35 47 \
5 --max-model-len 32000 \
6 --reasoning-parser qwen3 \
7 --language-model-only \
8 --kv-cache-dtype fp8 \
9 --max-num-batched-tokens 32768\
10 --tensor-parallel-size 4 \
11 --async-scheduling1# In virtual environment with speculators installed
2CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
3 --standalone \
4 --nproc_per_node 4 \
5 scripts/train.py \
6 --verifier-name-or-path Qwen/Qwen3.5-397B-A17B \
7 --data-path ./output \
8 --on-missing generate \
9 --on-generate delete \
10 --scheduler-type cosine \
11 --draft-vocab-size 24576 \
12 --max-anchors 3072 \
13 --target-layer-ids 3 15 23 35 47 \
14 --speculator-type dflash \
15 --logger trackio \
16 --lr 0.0006 \
17 --epochs 5 \
18 --draft-config 397_config.json| Base Model | Qwen/Qwen3.5-397B-A17B |
| Chat Template | Qwen/Qwen3.5-397B-A17B (use /chat/completions endpoint) |
| Format | Safetensors |
| License | Apache 2.0 |
| Validation Hardware | Nvidia H200 |
1# Install vLLM from the required PR
2pip install git+https://github.com/vllm-project/vllm.git
3
4# Deploy with speculative decoding
5vllm serve Qwen/Qwen3.5-397B-A17B \
6 --tensor-parallel-size 8 \
7 --max-num-batched-tokens 32768 \
8 --attention-backend FLASH_ATTN \
9 --speculative-config '{
10 "model": "RedHatAI/Qwen3.5-397B-A17B-speculator.dflash",
11 "num_speculative_tokens": 7,
12 "method": "dflash"
13 }'
14| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 83.9% | 68.6% | 56.6% | 46.9% | 39.1% | 32.4% | 26.2% | 4.54 |
| math_reasoning | 85.5% | 71.9% | 61.1% | 52.3% | 44.6% | 37.6% | 31.2% | 4.84 |
| qa | 69.8% | 47.3% | 33.1% | 24.5% | 18.8% | 14.0% | 10.1% | 3.18 |
| question | 73.6% | 52.0% | 37.6% | 28.6% | 22.7% | 18.0% | 13.9% | 3.46 |
| rag | 78.7% | 59.3% | 45.0% | 34.7% | 26.9% | 20.7% | 15.6% | 3.81 |
| summarization | 72.5% | 49.6% | 34.8% | 25.5% | 19.1% | 14.4% | 10.6% | 3.27 |
| tool_call | 73.9% | 52.2% | 36.7% | 26.2% | 18.8% | 13.3% | 9.7% | 3.31 |
| translation | 67.1% | 41.3% | 23.6% | 13.0% | 7.0% | 3.8% | 2.0% | 2.58 |
| writing | 73.6% | 52.0% | 37.6% | 28.6% | 22.6% | 18.0% | 13.9% | 3.46 |
