Converted with RKLLM Toolkit v1.2.3. This model re-ranks retrieved documents for RAG pipelines, improving result quality over pure vector-similarity search.

Key Details

Property	Value
Base Model	Qwen/Qwen3-Reranker-0.6B
Toolkit Version	RKLLM Toolkit v1.2.3
Runtime Version	RKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended)
Quantization	w8a8 (8-bit weights, 8-bit activations)
Target Platform	RK3588
NPU Cores	3
Optimization Level	1
Hybrid Ratio	0.5
Model Type	Reranker (cross-encoder)
Languages	English, Chinese (multilingual)

Why This Model?

A reranker is the second stage of a RAG retrieval pipeline — after the embedding model retrieves candidates, the reranker scores each (query, document) pair more precisely. Running it on the RK3588 NPU keeps the full pipeline local and GPU-free.

Pair with the Qwen3-Embedding-0.6B for a complete local retrieval stack.

Hardware Tested

Orange Pi 5 Plus — RK3588, 16GB RAM, Armbian Linux
RKNPU driver 0.9.8
RKLLM Runtime v1.2.3

Usage

With the RKLLM Reranker Service

bash

mkdir -p ~/models/Qwen3-Reranker-0.6B
cd ~/models/Qwen3-Reranker-0.6B
git lfs install && git clone https://huggingface.co/GatekeeperZA/Qwen3-Reranker-0.6B-RKLLM-v1.2.3 .

Use with GatekeeperZA/RKLLM-API-Server reranker endpoint.

File Listing

File	Description
`Qwen3-Reranker-0.6B-rk3588-w8a8-opt-1-hybrid-ratio-0.5.rkllm`	Quantized reranker model for RK3588 NPU

Compatibility Notes

Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
RKNPU driver: ≥ 0.9.6
SoCs: RK3588 / RK3588S. Not compatible with RK3576 without reconversion.
RAM: ~1GB loaded.

Acknowledgements

Alibaba Qwen Team for Qwen3-Reranker
Rockchip / airockchip for the RKLLM toolkit and runtime
Converted by GatekeeperZA