You can deploy this model and start earning money today!

adeepv/
Qwen3.8-27B-w8a8sc-Ascend310P

Views

No views yet

Qwen3.8-27B — W8A8SC for Ascend 310P (Atlas 300I Duo)

Sparse-compressed int8 (W8A8SC) build of Qwen3.8-27B for Ascend 310P (~10 GiB weights per rank). Serves the full 262,144 context on a single Atlas 300I Duo card (TP=2, needle verified at 225K), or fits on a single 310P die (TP=1) with ~32K working context. Layouts: v2/TP2 (two dies, one Duo card) and v2/TP1 (single die).

Why this exists: the published W8A8-dynamic builds crash on 310P with aicore error 507015 (broken tiling branches in QuantBatchMatmulV3_NZ_NZ_int8_int8_fp16 — deterministic per weight-shard shape, no safe chunk size exists for TP=2). W8A8SC uses a different, compressed-sparse kernel that is stable on this SoC.

Measured (on hardware, mask patches applied)

build	chips	KV capacity	working context	prefill	decode
v2 TP=2 (one Duo card)	2	291,667 tok	full 262,144 (needle @225,046 found, ingest 1051 s)	214 tok/s @225K	~4.5–5 tok/s
v2 TP=1 (single die)	1	36,352 tok	~31–32K (needle @30,898 found)	270–283 tok/s	5.29 / 5.13 tok/s (1K / 15K ctx)

TP=1 caveat: requests that land within a few hundred tokens of the KV ceiling queue forever (reproduced twice) — keep --max-model-len 32768 and prompts under ~31K.

Verified software versions

component	version
image	`quay.io/ascend/vllm-ascend:nightly-releases-v0.25.1rc-310p`
image digest	`sha256:e1bad30971ee0e0995d7511a39a224e4f62f6206f64de85a42513cf08d206880`
vLLM	0.25.1
torch / torch-npu	2.10.0 / 2.10.0.post2
CANN toolkit	9.1.0-beta.1
NPU driver	26.0.rc1 (firmware 9.0.0.0.205)

nightly-releases-* is a rolling family — pin the digest above.

Launch — v2, TP=2, full 256K context (one Duo card)

bash

docker run -d --name q38 --network host --shm-size 16gb --ipc host \
  --device /dev/davinci0 --device /dev/davinci1 \
  --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /path/to/this/repo:/model \
  -v /path/to/this/repo/patches/attention_mask.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/attention/attention_mask.py:ro \
  -v /path/to/this/repo/patches/metadata_builder.py:/vllm-workspace/vllm-ascend/vllm_ascend/_310p/attention/metadata_builder.py:ro \
  -v /path/to/this/repo/patches/attention_v1.py:/vllm-workspace/vllm-ascend/vllm_ascend/attention/attention_v1.py:ro \
  -e ASCEND_RT_VISIBLE_DEVICES=0,1 \
  --entrypoint bash quay.io/ascend/vllm-ascend@sha256:e1bad30971ee0e0995d7511a39a224e4f62f6206f64de85a42513cf08d206880 -c \
  'source /usr/local/Ascend/ascend-toolkit/set_env.sh && source /usr/local/Ascend/nnal/atb/set_env.sh && \
   exec vllm serve /model/v2/TP2 --host 0.0.0.0 --port 8100 \
     --tensor-parallel-size 2 --max-model-len 262144 --max-num-batched-tokens 4096 \
     --max-num-seqs 4 --gpu-memory-utilization 0.85 \
     --quantization ascend --load-format sharded_state \
     --dtype float16 --mamba-ssm-cache-dtype float16 \
     --no-enable-prefix-caching --enforce-eager --trust-remote-code \
     --limit-mm-per-prompt "{\"image\":0,\"video\":0}" \
     --additional-config "{\"ascend_compilation_config\":{\"fuse_norm_quant\":false}}"'

Launch — v2, TP=1, single 310P die

Same mounts and image; only one davinci device, TP=1, short context:

bash

  --device /dev/davinci0 ... \
  -e ASCEND_RT_VISIBLE_DEVICES=0 \
  ... exec vllm serve /model/v2/TP1 --host 0.0.0.0 --port 8100 \
     --tensor-parallel-size 1 --max-model-len 32768 --max-num-batched-tokens 4096 \
     --max-num-seqs 4 --gpu-memory-utilization 0.85 \
     --quantization ascend --load-format sharded_state \
     --dtype float16 --mamba-ssm-cache-dtype float16 \
     --no-enable-prefix-caching --enforce-eager --trust-remote-code \
     --limit-mm-per-prompt "{\"image\":0,\"video\":0}" \
     --additional-config "{\"ascend_compilation_config\":{\"fuse_norm_quant\":false}}"

Notes that matter:

patches/ is required for long context. Upstream sizes a dense O(L²) causal attention mask by max_model_len (256K → a 64 GiB allocation) and rebuilds it on every decode step. The three patched files size it to the actual batch and skip it in the states that never read it (on 310P only PrefillNoCache consumes it). Without them the practical ceiling is ~30–40K.
--max-num-batched-tokens trades prefill speed vs context ceiling: the splitfuse mask the kernel actually reads is [chunk × context]. 4096 reaches the full 262,144; 8192 prefills faster but tops out around ~100K.
--enforce-eager is mandatory on 310P; bf16 does not exist on this SoC.
Weights must be loaded with --load-format sharded_state from the matching layout directory (v2/TP2 or v2/TP1) — the compressed layout is per-rank; a TP=2 layout cannot be served with TP=1 or vice versa.
These shards contain the language model only: save_sharded_state exports the serving state dict, and with multimodal profiling disabled the vision tower is not part of it. Keep --limit-mm-per-prompt as shown.

Conversion recipe (how these weights were made)

Original Qwen/Qwen3.8-27B (BF16) → msmodelslim sparse quantization with the official qwen3-32b-w8a8s recipe (w_bit=4, a_bit=8, fraction=0.011, co_sparse=True, use_sigma=True, is_lowbit=True, mm_tensor=False, 64 samples of cn_en.jsonl, transformers 5.14.1, CPU fp32 calibration ≈ 50 min on 64 cores) → vLLM save_sharded_state + sparse compression (compress_excutor built from compress_graph sources).

Quantization skip-list (kept FLOAT): conv1d, in_proj_a/b, embeddings, lm_head, MTP head — everything else, including the GDN linear-attention projections and down_proj, is quantized. Markers: W8A8S: 1600, FLOAT: 784, W8A8SC: 401. Checkpoint ≈ 34 GB → ~10 GiB per TP=2 rank. (An earlier conservative build that kept GDN/down_proj/lm_head in fp16 was twice the size with a lower context ceiling and was removed.)

Quality: coherent generation and needle retrieval verified; no formal benchmark of the sparse quantization loss yet. The base recipe is identical to the official one for which ≤1.5pp degradation is reported on BoolQ/CEval/GSM8K.

中文简要：这是 Qwen3.8-27B 的 W8A8SC（稀疏压缩 int8）版本，每 rank 约 10 GiB——单张 Atlas 300I Duo（TP=2，v2/TP2）可跑完整 262,144 上下文 （needle 225K 实测通过，decode ~4.5–5 tok/s），或单颗 310P（TP=1，v2/TP1）跑 ~32K 上下文（decode ~5.3 tok/s）。W8A8-dynamic 在 310P 上会因 QuantBatchMatmulV3 内核 507015 崩溃，W8A8SC 走压缩内核路径，稳定。启动命令见上；patches/ 目录为长上下文所必需。镜像请按 digest 固定。

adeepv/Qwen3.8-27B-w8a8sc-Ascend310P