Views
No views yet
Qwen3_5ForConditionalGeneration) quantized to NVFP4
(ModelOpt mixed-precision mirroring RadixArk/Qwen3.8-27B-NVFP4's recipe,
FP8 KV cache) with one deliberate change: an unquantized bf16 lm_head
(DFlash2 refuses a quantized one — every public NVFP4 quant packs it to FP4),
plus baked-in dspark2 MTP draft tensors (mtp.* in model-mtp.safetensors,
referenced by model.safetensors.index.json).serving/. Stock vLLM can not serve this with
DFlash2 — it needs the pure-Python vLLM PR #52816 overlay plus a one-line
lm_head fix (see notes below), packaged here as two Dockerfiles.1# 0) Auth (private repo) — needs a perplexity-ai read token
2hf auth login
3
4# 1) Download this checkpoint + the DFlash2 draft (draft/) + the serving kit
5hf download perplexity-ai/pplx-computer-qwen-3-8-27b-dspark2-20260819 \
6 --local-dir ~/models/qwen38-dspark2-20260819
7
8# 2) Build the patched vLLM image (pure-Python patches, no kernel rebuild)
9cd ~/models/qwen38-dspark2-20260819/serving
10docker build -f Dockerfile -t vllm-dflash2:local .
11docker build -f Dockerfile.lmheadfix -t vllm-dflash2:lmheadfix .
12
13# 3) Serve
14MODEL_DIR=~/models/qwen38-dspark2-20260819 docker compose up -d
15
16# 4) Verify (first start compiles graphs — allow up to ~10 min)
17curl -sf localhost:8000/health && \
18curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
19 "model": "pplx-computer-qwen-3-8-27b-dspark2-20260819",
20 "messages": [{"role": "user", "content": "Say hi in five words."}],
21 "max_tokens": 64
22}' | python3 -m json.tooldocker compose up): GPU_UTIL (default 0.75 — raise
to ~0.9 on a dedicated box), MAX_LEN (262144), MAX_SEQS (4), SERVED_NAME
(the OpenAI-API model id; defaults to this repo's name — override if your
client expects a different id).1vllm serve /models/qwen38 \
2 --served-model-name pplx-computer-qwen-3-8-27b-dspark2-20260819 \
3 --host 0.0.0.0 --port 8000 \
4 --gpu-memory-utilization 0.75 \
5 --max-model-len 262144 \
6 --max-num-seqs 4 \
7 --max-num-batched-tokens 8192 \
8 --async-scheduling \
9 --speculative-config '{"method": "dflash", "model": "<model-dir>/draft", "num_speculative_tokens": 7}' \
10 --reasoning-parser qwen3 \
11 --tool-call-parser qwen3_coder \
12 --enable-auto-tool-choicedraft/ (a mirror of
incoai/Qwen3.8-27B-DFlash2,
2026-08-19) and referenced by local path in the compose — the repo is fully
self-contained, no external downloads at serve time. The draft is separate
from the target weights: the in-checkpoint mtp.* tensors are the dspark2
MTP module — MTP k=7 reaches similar acceptance but decodes ~2× slower
(sequential draft passes), so DFlash2 is the recommended path.lm_head; it is excluded via exclude_modules in both
hf_quant_config.json and config.json (vLLM's modelopt loader keys on
the exclude wildcards, not on absence from quantized_layers).Dockerfile.lmheadfix: vLLM gives an excluded lm_head an
UnquantizedLinearMethod, which PR #52816's isinstance check wrongly
rejects (it only accepts UnquantizedEmbeddingMethod). The second image
layer relaxes that check.--max-num-batched-tokens 8192 is required: with spec decode enabled,
vLLM otherwise clamps scheduling to 2048 and prefill collapses.