Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3-Reranker-0.6B
|
| 4 |
+
tags:
|
| 5 |
+
- rkllm
|
| 6 |
+
- rk3588
|
| 7 |
+
- rockchip
|
| 8 |
+
- npu
|
| 9 |
+
- quantized
|
| 10 |
+
- qwen3
|
| 11 |
+
- reranker
|
| 12 |
+
- text-ranking
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
+
- zh
|
| 16 |
+
pipeline_tag: text-ranking
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# Qwen3-Reranker-0.6B — RKLLM v1.2.3 (w8a8, RK3588)
|
| 20 |
+
|
| 21 |
+
RKLLM conversion of [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) for Rockchip RK3588 NPU inference.
|
| 22 |
+
|
| 23 |
+
Converted with RKLLM Toolkit v1.2.3. This model re-ranks retrieved documents for RAG pipelines, improving result quality over pure vector-similarity search.
|
| 24 |
+
|
| 25 |
+
## Key Details
|
| 26 |
+
|
| 27 |
+
| Property | Value |
|
| 28 |
+
|----------|-------|
|
| 29 |
+
| Base Model | Qwen/Qwen3-Reranker-0.6B |
|
| 30 |
+
| Toolkit Version | RKLLM Toolkit v1.2.3 |
|
| 31 |
+
| Runtime Version | RKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended) |
|
| 32 |
+
| Quantization | w8a8 (8-bit weights, 8-bit activations) |
|
| 33 |
+
| Target Platform | RK3588 |
|
| 34 |
+
| NPU Cores | 3 |
|
| 35 |
+
| Optimization Level | 1 |
|
| 36 |
+
| Hybrid Ratio | 0.5 |
|
| 37 |
+
| Model Type | Reranker (cross-encoder) |
|
| 38 |
+
| Languages | English, Chinese (multilingual) |
|
| 39 |
+
|
| 40 |
+
## Why This Model?
|
| 41 |
+
|
| 42 |
+
A reranker is the second stage of a RAG retrieval pipeline — after the embedding model retrieves candidates, the reranker scores each (query, document) pair more precisely. Running it on the RK3588 NPU keeps the full pipeline local and GPU-free.
|
| 43 |
+
|
| 44 |
+
Pair with the Qwen3-Embedding-0.6B for a complete local retrieval stack.
|
| 45 |
+
|
| 46 |
+
## Hardware Tested
|
| 47 |
+
|
| 48 |
+
- **Orange Pi 5 Plus** — RK3588, 16GB RAM, Armbian Linux
|
| 49 |
+
- RKNPU driver 0.9.8
|
| 50 |
+
- RKLLM Runtime v1.2.3
|
| 51 |
+
|
| 52 |
+
## Usage
|
| 53 |
+
|
| 54 |
+
### With the RKLLM Reranker Service
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
mkdir -p ~/models/Qwen3-Reranker-0.6B
|
| 58 |
+
cd ~/models/Qwen3-Reranker-0.6B
|
| 59 |
+
git lfs install && git clone https://huggingface.co/GatekeeperZA/Qwen3-Reranker-0.6B-RKLLM-v1.2.3 .
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
Use with [GatekeeperZA/RKLLM-API-Server](https://github.com/GatekeeperZA/RKLLM-API-Server) reranker endpoint.
|
| 63 |
+
|
| 64 |
+
## File Listing
|
| 65 |
+
|
| 66 |
+
| File | Description |
|
| 67 |
+
|------|-------------|
|
| 68 |
+
| `Qwen3-Reranker-0.6B-rk3588-w8a8-opt-1-hybrid-ratio-0.5.rkllm` | Quantized reranker model for RK3588 NPU |
|
| 69 |
+
|
| 70 |
+
## Compatibility Notes
|
| 71 |
+
|
| 72 |
+
- Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
|
| 73 |
+
- RKNPU driver: ≥ 0.9.6
|
| 74 |
+
- SoCs: RK3588 / RK3588S. Not compatible with RK3576 without reconversion.
|
| 75 |
+
- RAM: ~1GB loaded.
|
| 76 |
+
|
| 77 |
+
## Acknowledgements
|
| 78 |
+
|
| 79 |
+
- Alibaba Qwen Team for Qwen3-Reranker
|
| 80 |
+
- Rockchip / airockchip for the RKLLM toolkit and runtime
|
| 81 |
+
- Converted by [GatekeeperZA](https://huggingface.co/GatekeeperZA)
|