Runs on a 16 GB Radxa ROCK 5B+: 20.4 tok/s prefill, 4.8 tok/s decode on a 1967-token prompt — the fastest prefill we have measured on this hardware, and ahead of the public quant on every axis while being 2.1 GiB smaller.


file	`Q35-npuaware.gguf`
size	14.89 GiB (3.60 bits/weight)
source	`bartowski/Qwen_Qwen3.6-35B-A3B-GGUF` Q4_0
runtime	`rk-llama.cpp` with the RKNPU2 backend (rknpu 0.9.8 / librknnrt 2.3.x)
architecture	hybrid — 30 Gated-DeltaNet/SSM blocks + 10 full-attention + 1 MTP

Why a special quant for this chip

The RKNPU2 backend places a weight on the NPU only if its GGUF type is one of F16, Q8_0, Q6_K, Q4_0. Everything else runs on the CPU. RK3588's matmul also requires symmetric A/B precision — there is no W4A8 — so any NPU-resident weight is read as int8 regardless of how it is stored.

So a tensor you want the NPU to accelerate (prefill, compute-bound) must be one of those four types, while a tensor you want read cheaply (decode, byte-bound) must be a non-eligible type so it stays on CPU at its real width. Prefill and decode want opposite placement, and a quant choice becomes a placement choice.

What is inside

Only the expert tensors were changed. Every attention and SSM projection is copied bit-exact from the source, so all of them remain NPU-eligible and the NPU continues to carry prefill.

tensors	count	type	runs on
`ffn_up_exps`, `ffn_gate_exps`, `ffn_down_exps`	123	Q3_K — the only change	CPU
`attn_qkv`, `attn_gate` (the 30 recurrent blocks)	60	Q4_0, unchanged	NPU
`attn_k`, `attn_v`	22	Q8_0, unchanged	NPU
`attn_output`	11	Q6_K	NPU
`ssm_out`	30	Q4_0 / Q8_0, unchanged	NPU
`ffn_*_shexp` (shared experts, read every token)	63	Q8_0, unchanged	NPU
`ssm_*`, norms, routers	—	F32, unchanged	CPU

Type mix: 368 × F32 · 123 × Q3_K · 101 × Q8_0 · 83 × Q4_0 · 16 × Q6_K · 2 × BF16.

Performance

Radxa ROCK 5B+ · RK3588 · 16 GB LPDDR5 · 4× Cortex-A76 @ 2352 MHz (pinned, performance governor) · NVMe. 1967-token prompt, 150 output tokens, prompt caching disabled, warm.

quant	size	prefill tok/s	decode tok/s	turn time
this build	14.89 GiB	20.4	4.8	128 s
`UD-IQ4_XS`	16.96 GiB	15.3	4.0	166 s

Better on every axis: prefill +33 %, decode +20 %, turn time −23 %, in a file 2.1 GiB smaller. A live 2477-token request on a cold start measured 21.5 tok/s prefill.

Choosing this model: long prompts, not long conversations

30 of the 41 blocks are Gated-DeltaNet. Recurrent state is a running summary, so it cannot be rewound to an arbitrary position the way a KV cache can. A follow-up question in the same conversation therefore re-processes most of the thread — around 94 % of the context, roughly 20 s to first token on turn two, growing with thread length.

That is inherent to the architecture rather than a property of this quantisation. It makes the model an excellent fit for long single prompts — documents, code, logs, retrieved context — and a poor fit for long back-and-forth chat, where a non-recurrent model will feel far more responsive.

Usage

Needs a rk-llama.cpp build with the RKNPU2 backend.

bash

export RKNPU_HYBRID=W8A8_STANDARD RKNPU_GLUE=1
export LLAMA_RECURRENT_ON_CPU=1
ulimit -n 1000000

taskset -c 4-7 ./llama-server \
  -m Q35-npuaware.gguf \
  -ngl 99 --cpu-moe --no-repack \
  -np 1 -t 4 --jinja --no-warmup \
  -c 32768 -fa on -cram 2048 \
  --host 0.0.0.0 --port 8095

LLAMA_RECURRENT_ON_CPU=1 is essential for this model. It assigns the recurrent layers to the CPU device, which is what allows the fused Gated-DeltaNet kernels to stay enabled. Without it both fused paths are disabled silently — no error, just a large slowdown. Check the load log for fused Gated Delta Net … enabled; it is the most important line for this architecture. (This variable is provided by the fork linked below.)
RKNPU_HYBRID=W8A8_STANDARD — selects the int8 pipeline; without it eligible weights fall to the int4 path, which is slower and worse.
-fa on here, unlike on a pure-attention model: only 10 of 41 blocks attend, so the fused kernel is roughly neutral and on edges ahead.
-c 32768 is inexpensive on this architecture — KV costs about 20 KiB/token, versus roughly 96 KiB/token on a comparable dense-attention MoE.
--cpu-moe --no-repack — keeps experts memory-mapped as reclaimable page cache, which is what lets a model larger than board RAM run. --repack is on by default and will exhaust memory on a large MoE.
ulimit -n 1000000 — required; each matmul context imports a DMA file descriptor and the default 1024 limit is exhausted.
taskset -c 4-7 -t 4 — A76 cores only; including the A55s costs about half the throughput.

Optional: fewer active experts

Adding --override-kv qwen35moe.expert_used_count=int:4 routes 4 of 256 experts per token instead of the trained 8. On this model the trained top-8 setting costs roughly 44 % of decode speed, so the override is a large throughput win — but it is a real quality trade, so it is left out of the command above.

Reproducing

Because only the experts change, a full per-tensor type file is used so every other tensor is copied rather than requantised. llama-quantize copies a tensor when its target type already equals its current type, so listing every tensor at its current type makes each one a no-op:

bash

# every tensor at its current type, except the three expert groups -> q3_K
python3 gen_ttypes.py \
  --targets=ffn_up_exps.weight,ffn_gate_exps.weight,ffn_down_exps.weight \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf tt35.txt q3_K

llama-quantize --allow-requantize --tensor-type-file tt35.txt \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf Q35-npuaware.gguf Q4_0 8

gen_ttypes.py is in the fork linked below. Worth checking the resulting tensor types: llama.cpp's mixture heuristic can promote a tensor to a higher type than requested.

Notes

Tuned for a 16 GB RK3588; on a 32 GB board the balance shifts.
This tokenizer produces roughly 6 % more tokens for the same text than Qwen3-30B (vocabulary 248,320 vs 151,936), which is worth remembering when comparing tokens/s figures between the two.
Deliberately not a general-purpose quant — the type choices only pay off given the NPU eligibility rules above.
Optimise for throughput rather than NPU utilisation. Prefill runs at low NPU occupancy while the CPU is saturated, yet moving attention off the NPU still costs a large share of prefill speed.

Thanks and credits

Sincere thanks to the Qwen team at Alibaba for releasing Qwen3.6-35B-A3B under Apache-2.0 — a remarkable hybrid architecture, and the kind of openness that makes edge work like this possible.

Thanks also to bartowski, whose Qwen_Qwen3.6-35B-A3B-GGUF conversion was the starting point for this build.

llama.cpp — ggml-org and its contributors (MIT)
RKNPU2 ggml backend — invisiofficial/rk-llama.cpp, which made RK3588 NPU inference possible in the first place
Rockchip for the RKNN runtime, and Radxa for the ROCK 5B+

Quantisation and RK3588 tuning: Mojo24x7/rk-llama.cpp

Mojo24x7/Qwen3.6-35B-A3B-npuaware-rk3588-GGUF

Qwen3.6-35B-A3B — NPU-aware GGUF for Rockchip RK3588