license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: rkllm
pipeline_tag: text-generation
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- qwen
- qwen2.5
Qwen2.5-1.5B-Instruct-RKLLM
RKLLM-converted versions of Qwen2.5-1.5B-Instruct for Rockchip RK3576 and RK3588.
.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.
Base Model
- Model: Qwen2.5-1.5B-Instruct
- Author: Alibaba Cloud Qwen Team
- Original Model: Qwen/Qwen2.5-1.5B-Instruct
- Original License: Apache-2.0
Supported Platforms
| SoC | Status |
|---|---|
| RK3576 | ✅ Tested |
| RK3588 | ✅ Supported (not tested) |
RK3576 has been tested. RK3588 is supported as a target platform but has not been board-tested because no RK3588 board is available.
Conversion
- RKLLM Toolkit: 1.2.3
- Target Platforms: RK3576, RK3588
Available Variants
| File | Quantization |
|---|---|
| Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm | W4A16 |
| Qwen2.5-1.5B-Instruct_w8a8_RK3576.rkllm | W8A8 |
| Qwen2.5-1.5B-Instruct_w8a8_RK3588.rkllm | W8A8 |
Repository Structure
.
├── RK3576
│ ├── Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm
│ └── Qwen2.5-1.5B-Instruct_w8a8_RK3576.rkllm
└── RK3588
└── Qwen2.5-1.5B-Instruct_w8a8_RK3588.rkllm
Usage
Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching RK3576 or RK3588 platform.
For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.
Docker Deployment
Use the model-specific image for the fastest deployment. The model is included in the image, so no separate model download is required:
sudo docker run --rm -it \
--privileged \
-p 8001:8001 \
-v /dev:/dev \
ghcr.io/hanzo-huang/rkllm-docker/qwen2.5-1.5b-instruct:w4a16-rk3576
Alternatively, use the reusable RKLLM environment image and mount a downloaded model:
sudo docker run --rm -it \
--privileged \
-p 8001:8001 \
-v /dev:/dev \
-v ./:/app/models:ro \
-e MODEL_PATH=/app/models/RK3576/Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm \
-e TARGET_PLATFORM=rk3576 \
ghcr.io/hanzo-huang/rkllm-docker:env-latest
The second command requires downloading the selected .rkllm file first and setting MODEL_PATH to its mounted path. Use the matching image, model variant, and TARGET_PLATFORM for the target Rockchip SoC.
Acknowledgements
This repository contains converted RKLLM versions of Qwen2.5-1.5B-Instruct.
Thanks to the Alibaba Cloud Qwen Team for releasing the original model.