You can deploy this model and start earning money today!

ccarpinteri/
Qwen3-4B-rk3588-rkllm-1.3.0

Name: Qwen3-4B-rk3588-rkllm-1.3.0
Availability: InStock

Views

No views yet

Qwen3-4B — RKLLM 1.3.0 / RK3588

Qwen/Qwen3-4B converted to RKLLM 1.3.0 format for the Rockchip RK3588 NPU.

Details


Base model	Qwen/Qwen3-4B
Quantization	W8A8 (8-bit weights + 8-bit activations)
RKLLM version	1.3.0
Target SoC	RK3588 (3 NPU cores)
File size	4.6 GB
Converted	2026-06-23

Usage with rkllama

rkllama provides an Ollama-compatible API for running RKLLM models on RK3588 devices.

Pull and run

bash

# Pull via rkllama (coming soon — manual install below in the meantime)
rkllama pull ccarpinteri/Qwen3-4B-rk3588-rkllm-1.3.0

Manual install

Download Qwen3-4B_W8A8_RK3588.rkllm from this repo
Place it at /path/to/rkllama/models/qwen3:4b/Qwen3-4B_W8A8_RK3588.rkllm
Create a Modelfile alongside it:

FROM="Qwen3-4B_W8A8_RK3588.rkllm"
HUGGINGFACE_PATH="Qwen/Qwen3-4B"
SYSTEM="You are a helpful assistant."
TEMPERATURE=0.7

Test

bash

curl http://localhost:11434/api/generate \
  -d '{"model":"qwen3:4b","prompt":"/no_think Hello! What can you help me with?","stream":false}'

Notes

Use /no_think prefix for fast chat mode; /think to enable chain-of-thought reasoning
Tested on a Rock 5 ITX (RK3588) running rkllama v0.0.73
~5.9 tokens/sec in chat mode on RK3588 NPU

Conversion

Converted using rkllm-toolkit 1.3.0 on an AWS EC2 r6a.xlarge (32GB RAM + 16GB swap):

python

from rkllm.api import RKLLM

llm = RKLLM()
llm.load_huggingface(model="./Qwen3-4B", device="cpu")
llm.build(
    do_quantization=True,
    optimization_level=1,
    quantized_dtype="w8a8",
    quantized_algorithm="normal",
    target_platform="rk3588",
    num_npu_core=3,
)
llm.export_rkllm("./Qwen3-4B_W8A8_RK3588.rkllm")