Runs comfortably on a 16 GB Radxa ROCK 5B+: 18.7 tok/s prefill, 4.9 tok/s decode on an 1855-token prompt — faster than any public quant we tested on this hardware, in a smaller file.


file	`Q30-npuaware.gguf`
size	13.72 GiB (3.86 bits/weight)
source	`Qwen3-30B-A3B-Instruct-2507` Q8_0
runtime	`rk-llama.cpp` with the RKNPU2 backend (rknpu 0.9.8 / librknnrt 2.3.x)

Why a special quant for this chip

The RKNPU2 backend places a weight on the NPU only if its GGUF type is one of F16, Q8_0, Q6_K, Q4_0. Everything else (Q3_K, Q4_K, Q5_K, IQ*) runs on the CPU. RK3588's matmul also requires symmetric A/B precision — there is no W4A8 — so any NPU-resident weight is read as int8 regardless of how it is stored.

That produces a sharp trade-off:

A tensor you want the NPU to accelerate — which helps prefill, since prefill is compute-bound — must be one of those four types.
A tensor you want read cheaply — which helps decode, since decode is byte-bound — must be a non-eligible type, so it stays on the CPU and is read at its real width instead of being inflated to int8.

Prefill and decode want opposite placement. A quant choice on this chip is therefore a placement choice, and general-purpose quants land on one side of the line by coincidence:

public quant	placement	outcome on RK3588
`Q4_0`	all eligible → NPU	good prefill; attention upcast to int8, so decode pays ~2× the bytes
`Q3_K_M`	none eligible → all CPU	good decode; NPU barely used and prefill drops sharply

This build assigns each group to the engine that suits it.

What is inside

tensors	type	runs on	reason
`attn_q`, `attn_k`, `attn_v`, `attn_output` (192)	Q4_0	NPU	eligible, so the NPU accelerates prefill
`output.weight`, `token_embd`	Q4_0	NPU	eligible
`ffn_up_exps`, `ffn_gate_exps`	Q3_K	CPU	non-eligible → read at real width; experts run on CPU under `--cpu-moe` anyway
`ffn_down_exps`	Q4_K	CPU	kept higher by llama.cpp's mixture heuristic, which protects the sensitive down-projection
norms, router	F32	CPU	unchanged

Type mix: 241 × F32 · 194 × Q4_0 · 96 × Q3_K · 45 × Q4_K · 3 × Q5_K.

Performance

Radxa ROCK 5B+ · RK3588 · 16 GB LPDDR5 · 4× Cortex-A76 @ 2352 MHz (pinned, performance governor) · NVMe. 1855-token prompt, 150 output tokens, prompt caching disabled, warm.

quant	size	prefill tok/s	decode tok/s	turn time
this build	13.72 GiB	18.7	4.9	130 s
`Q4_0`	16.04 GiB	17.8	4.2	140 s
`Q3_K_M`	13.70 GiB	13.4	5.8	165 s

Against Q4_0 it improves every axis — prefill +5 %, decode +16 %, turn time −7 % — and is 2.3 GiB smaller. Against Q3_K_M it gains +40 % prefill and −21 % turn time for 16 % less decode; the crossover is around output ≈ 0.6 × prompt tokens, so Q3_K_M only leads for short prompts with very long answers.

Multi-turn behaviour is strong: a follow-up question re-processes only 21 of 553 context tokens, giving 2.0 s to first token on turn two.

Usage

Needs a rk-llama.cpp build with the RKNPU2 backend.

bash

export RKNPU_HYBRID=W8A8_STANDARD RKNPU_GLUE=1
ulimit -n 1000000

taskset -c 4-7 ./llama-server \
  -m Q30-npuaware.gguf \
  -ngl 99 --cpu-moe --no-repack \
  -np 1 -t 4 --jinja --no-warmup \
  -c 16384 -fa off -cram 2048 \
  --host 0.0.0.0 --port 8095

A few flags are worth explaining:

RKNPU_HYBRID=W8A8_STANDARD — selects the int8 pipeline. Without it, Q4_0 weights default to the int4 W4A4_HADAMARD path, which is both slower on expert matmuls and noticeably worse in quality.
ulimit -n 1000000 — required. Every matmul context imports a DMA file descriptor; the default 1024 limit is exhausted and the process crashes.
--cpu-moe --no-repack — keeps expert weights memory-mapped as reclaimable page cache, which is what allows a model larger than board RAM to run. --repack is on by default and will exhaust memory on a large MoE.
taskset -c 4-7 -t 4 — the four Cortex-A76 cores only. Including the A55 cores costs about half the throughput; oversubscribing the A76s costs about a quarter.
-fa off — at longer contexts the decomposed attention path outperforms ggml's fused CPU kernel here: about +22 % decode at 1574 tokens, for ~1.5 % less prefill.
-cram 2048 — bounds the prompt cache so it cannot evict the expert pages this model depends on.

Optional: fewer active experts

Adding --override-kv qwen3moe.expert_used_count=int:4 routes 4 experts per token instead of the trained 8, which gives roughly +30 % decode. It is a genuine quality trade (measurably higher perplexity), so it is left out of the command above — enable it only if throughput matters more than fidelity for your use case.

Reproducing

With llama-quantize, from the Q8_0:

bash

llama-quantize --allow-requantize \
  --tensor-type attn_q=q4_0 --tensor-type attn_k=q4_0 \
  --tensor-type attn_v=q4_0 --tensor-type attn_output=q4_0 \
  --output-tensor-type q4_0 --token-embedding-type q4_0 \
  Qwen3-30B-A3B-Instruct-2507-Q8_0.gguf Q30-npuaware.gguf Q3_K_M 8

About 7 minutes on the board itself. Note that --tensor-type is a request rather than a command — llama.cpp's k-quant mixture heuristic may promote some tensors to a higher type — so it is worth dumping the tensor types of the result to confirm what you got.

Notes

Tuned for a 16 GB RK3588. On a 32 GB board more of the model stays resident and the balance shifts.
This is deliberately not a general-purpose quant. The type choices only pay off given the NPU eligibility rules above; on a GPU they would cost quality for no benefit.
On this hardware, throughput is the metric to optimise, not NPU utilisation. Prefill runs with the NPU at low occupancy while the CPU is saturated, yet moving attention off the NPU still costs around 37 % of prefill speed.

Thanks and credits

Sincere thanks to the Qwen team at Alibaba for releasing Qwen3-30B-A3B-Instruct-2507 under Apache-2.0 — an outstanding model, and the openness that makes work like this possible.

llama.cpp — ggml-org and its contributors (MIT)
RKNPU2 ggml backend — invisiofficial/rk-llama.cpp, which made RK3588 NPU inference possible in the first place
Rockchip for the RKNN runtime, and Radxa for the ROCK 5B+

Quantisation and RK3588 tuning: Mojo24x7/rk-llama.cpp

Mojo24x7/Qwen3-30B-A3B-npuaware-rk3588-GGUF

Qwen3-30B-A3B-Instruct-2507 — NPU-aware GGUF for Rockchip RK3588