| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-1.5B-Instruct |
| tags: |
| - rkllm |
| - rk3588 |
| - npu |
| - quantized |
| - w8a8 |
| pipeline_tag: text-generation |
| --- |
| |
| # Qwen2.5-1.5B-Instruct for RKLLM v1.2.3 (RK3588 NPU) |
|
|
| Pre-converted Qwen2.5-1.5B-Instruct for the Rockchip RK3588 NPU using rknn-llm runtime v1.2.3. |
|
|
| Runs on Orange Pi 5 Plus, Rock 5B, Radxa NX5, and other RK3588-based SBCs with 8GB+ RAM. |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| |------|------|-------------| |
| | `Qwen2.5-1.5B-Instruct-rk3588-v123-w8a8.rkllm` | 2.0 GB | LLM (W8A8 quantized, 8192 token context) | |
|
|
| ## Quick Start |
|
|
| ### Directory Structure |
|
|
| ``` |
| ~/models/Qwen2.5-1.5B-Instruct/ |
| Qwen2.5-1.5B-Instruct-rk3588-v123-w8a8.rkllm |
| ``` |
|
|
| ### Using with RKLLM API Server |
|
|
| This model is designed for use with the [RKLLM API Server](https://github.com/airockchip/rknn-llm), which provides an OpenAI-compatible API for RK3588 NPU inference. The server auto-discovers `.rkllm` files by scanning subdirectories of your models folder. |
|
|
| ```bash |
| # Place the model in your models directory |
| mkdir -p ~/models/Qwen2.5-1.5B-Instruct |
| # Copy .rkllm file here — the API server will find it automatically |
| sudo systemctl restart rkllm-api |
| ``` |
|
|
| The model will appear as `qwen2.5-1.5b-instruct` in the OpenAI-compatible model list. |
|
|
| ## Export Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Source | [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) | |
| | Tool | rkllm-toolkit v1.2.3 | |
| | Quantization | W8A8 (8-bit weights, 8-bit activations) | |
| | Optimization level | 1 | |
| | Target platform | rk3588 | |
| | NPU cores | 3 | |
| | Max context | 8192 tokens | |
|
|
| ## Performance Benchmarks |
|
|
| Tested on Orange Pi 5 Plus (16GB RAM), RK3588 SoC, RKNPU driver 0.9.8: |
|
|
| | Metric | Value | |
| |--------|-------| |
| | Decode speed | ~19 tok/s | |
| | Model load time | ~3 s | |
| | Peak RAM | ~2.2 GB | |
|
|
| ## Known Limitations |
|
|
| - **Tool calling**: Qwen2.5-1.5B is a general-purpose instruct model, not fine-tuned for structured function calling. For agent/tool-calling workloads, see [GatekeeperZA/xLAM-1b-fc-r-RKLLM-v1.2.3](https://huggingface.co/GatekeeperZA/xLAM-1b-fc-r-RKLLM-v1.2.3). |
| - **Context**: Compiled at 8192 tokens. The runtime will truncate inputs that exceed this limit. |
| - **Version lock**: This file requires RKLLM runtime v1.2.3 and NPU driver v0.9.8. It is **not compatible** with v1.3.0 or later runtimes. |
|
|
| ## License |
|
|
| Apache 2.0, inherited from Qwen2.5-1.5B-Instruct. |
|
|
| ## Credits |
|
|
| - Model: [Qwen Team](https://huggingface.co/Qwen) for Qwen2.5-1.5B-Instruct |
| - Runtime: [Rockchip / airockchip](https://github.com/airockchip/rknn-llm) for rkllm-toolkit |
| - API Server: [RKLLM API Server](https://github.com/airockchip/rknn-llm) — OpenAI-compatible server for RK3588 NPU |
|
|
