| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-4B-Instruct-2507 |
| tags: |
| - rkllm |
| - rk3588 |
| - rockchip |
| - npu |
| - quantized |
| - qwen3 |
| language: |
| - en |
| - zh |
| --- |
| |
| # Qwen3-4B-Instruct-2507 — RKLLM v1.2.3 (w8a8, RK3588) |
|
|
| RKLLM conversion of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) for Rockchip RK3588 NPU inference. |
|
|
| Converted with RKLLM Toolkit v1.2.3. The `2507` suffix denotes the July 2025 refresh of Qwen3-4B with improved instruction following. |
|
|
| > **Note:** This model does **not** produce `<think>` reasoning blocks in this RKLLM build (thinking mode is disabled). See [RKLLM thinking mode limitations](https://github.com/airockchip/rknn-llm). |
|
|
| ## Key Details |
|
|
| | Property | Value | |
| |----------|-------| |
| | Base Model | Qwen/Qwen3-4B-Instruct-2507 | |
| | Toolkit Version | RKLLM Toolkit v1.2.3 | |
| | Runtime Version | RKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended) | |
| | Quantization | w8a8, group size 128 | |
| | Target Platform | RK3588 | |
| | NPU Cores | 3 | |
| | Max Context Length | 16384 tokens | |
| | Optimization Level | 1 | |
| | Hybrid Ratio | 0.0 | |
| | Thinking Mode | ❌ Disabled | |
| | Languages | English, Chinese (multilingual) | |
|
|
| ## Why This Model? |
|
|
| Qwen3-4B-Instruct-2507 is the July 2025 update to Alibaba's Qwen3-4B, with improved reasoning and instruction following. At 4B parameters it is the largest text-only model in this RK3588 lineup and handles complex prompts well despite the quantization. |
|
|
| The 16k context window (vs 8k on smaller models) enables long document summarisation and multi-turn conversations. |
|
|
| ## Hardware Tested |
|
|
| - **Orange Pi 5 Plus** — RK3588, 16GB RAM, Armbian Linux |
| - RKNPU driver 0.9.8 |
| - RKLLM Runtime v1.2.3 |
|
|
| ## Usage |
|
|
| ### With the RKLLM API Server |
|
|
| ```bash |
| mkdir -p ~/models/Qwen3-4B-Instruct-2507 |
| cd ~/models/Qwen3-4B-Instruct-2507 |
| git lfs install && git clone https://huggingface.co/GatekeeperZA/Qwen3-4B-Instruct-2507-RKLLM-v1.2.3 . |
| ``` |
|
|
| Use with [GatekeeperZA/RKLLM-API-Server](https://github.com/GatekeeperZA/RKLLM-API-Server). |
|
|
| ### With the official RKLLM API demo |
|
|
| ```bash |
| git clone https://github.com/airockchip/rknn-llm.git |
| cd rknn-llm/examples/rkllm_api_demo |
| ./build/rkllm_api_demo /path/to/Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllm 8192 16384 |
| ``` |
|
|
| ## File Listing |
|
|
| | File | Description | |
| |------|-------------| |
| | `Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllm` | Quantized model for RK3588 NPU | |
|
|
| ## Compatibility Notes |
|
|
| - Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended. |
| - RKNPU driver: ≥ 0.9.6 |
| - SoCs: RK3588 / RK3588S. Not compatible with RK3576 without reconversion. |
| - RAM: ~5.5GB loaded. Requires 8GB+ board (16GB recommended). |
|
|
| ## Acknowledgements |
|
|
| - Alibaba Qwen Team for Qwen3 |
| - Rockchip / airockchip for the RKLLM toolkit and runtime |
| - Converted by [GatekeeperZA](https://huggingface.co/GatekeeperZA) |
|
|
