Views
No views yet
Weights are int8, activations stay FP16. This is not a speed optimization on 310P — it is a memory optimization: halving the weights frees NPU memory for a much larger KV cache, enabling up to 256K context on 4× 310P3 chips (TP=4).
QuantBatchMatmulV3 kernel, L0C conflict).| FP16 27B | W8A16 27B (this) | |
|---|---|---|
| Weights | ~54 GB | ~27 GB |
| Context (TP=4) | KV-starved | 256K (KV ~832K tokens) |
| Decode (4× 310P3) | ~3.8 t/s | ~3.1 t/s |
| Quality | reference | ≈lossless (data-free RTN int8) |
msmodelslim data-free RTN int8 (ascend modelslim ascendV1 format), via
AutoModelForImageTextToText (keeps the multimodal wrapper; visual.* left unquantized).
See recipe/ for the conversion notes.npu_weight_quant_batchmatmul expects weights in ND [K,N] layout, not FRACTAL_NZ
(NZ → task not supported). Stock w8a16.py casts to NZ → fails. This repo ships a 310P scheme
override (patches/methods_init_310.py) that transposes weights to contiguous ND without
maybe_trans_nz. You must mount the patches (below) — otherwise loading fails with a
k-mismatch / task not supported error.quay.io/ascend/vllm-ascend:nightly-main-310p.1# 1) get weights + patches
2huggingface-cli download adeepv/Qwen3.6-27B-W8A16-Ascend310P --local-dir /opt/vllm/models/qwen36-w8a16
3
4# 2) run (patches mounted into the image, --enforce-eager is mandatory on 310P)
5docker run -d --name vllm-qwen36-w8a16 --network host --restart unless-stopped \
6 --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 \
7 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc \
8 -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
9 -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
10 -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
11 -v /etc/ascend_install.info:/etc/ascend_install.info \
12 -v /usr/local/dcmi:/usr/local/dcmi \
13 -v /opt/vllm/models:/root/.cache/huggingface \
14 -v /opt/vllm/models/qwen36-w8a16/patches/methods_init_310.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/quantization/methods/__init__.py \
15 -v /opt/vllm/models/qwen36-w8a16/patches/gdn_310.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/ops/fla/gdn_310.py \
16 -v /opt/vllm/models/qwen36-w8a16/patches/chunk_gated_delta_rule.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/ops/fla/chunk_gated_delta_rule.py \
17 -v /opt/vllm/models/qwen36-w8a16/patches/attention_mask.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/attention/attention_mask.py \
18 -v /opt/vllm/models/qwen36-w8a16/patches/attention_v1_generic.py:/vllm-workspace/vllm-ascend/vllm_ascend/attention/attention_v1.py \
19 -v /opt/vllm/models/qwen36-w8a16/patches/modelslim_config.py:/vllm-workspace/vllm-ascend/vllm_ascend/quantization/modelslim_config.py \
20 --entrypoint bash quay.io/ascend/vllm-ascend:nightly-main-310p -c '
21 source /usr/local/Ascend/ascend-toolkit/set_env.sh &&
22 source /usr/local/Ascend/cann-9.1.0-beta.1/share/info/ascendnpu-ir/bin/set_env.sh &&
23 source /usr/local/Ascend/nnal/atb/set_env.sh &&
24 exec vllm serve /root/.cache/huggingface/qwen36-w8a16 \
25 --host 0.0.0.0 --port 8000 \
26 --tensor-parallel-size 4 --dtype float16 --enforce-eager \
27 --max-model-len 262144 --gpu-memory-utilization 0.9 \
28 --no-enable-prefix-caching --served-model-name qwen3.6-27b \
29 --quantization ascend --language-model-only --trust-remote-code \
30 --reasoning-parser qwen3'--quantization ascend selects the ascend W8A16 scheme; --enforce-eager is required
(aclgraph fails on 310P at AclmdlRICaptureEnd, error 507903).--tensor-parallel-size 2 and a smaller --max-model-len.reasoning_content; pair with --reasoning-parser qwen3.LICENSE.