Qwen2.5-1.5B-Instruct-RKLLM

RKLLM-converted versions of Qwen2.5-1.5B-Instruct for Rockchip RK3576 and RK3588.

.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.

Base Model

• Model: Qwen2.5-1.5B-Instruct
• Author: Alibaba Cloud Qwen Team
• Original Model: Qwen/Qwen2.5-1.5B-Instruct
• Original License: Apache-2.0

Supported Platforms

SoC	Status
RK3576	✅ Tested
RK3588	✅ Supported (not tested)

RK3576 has been tested. RK3588 is supported as a target platform but has not been board-tested because no RK3588 board is available.

Conversion

• RKLLM Toolkit: 1.2.3
• Target Platforms: RK3576, RK3588

Available Variants

File	Quantization
Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm	W4A16
Qwen2.5-1.5B-Instruct_w8a8_RK3576.rkllm	W8A8
Qwen2.5-1.5B-Instruct_w8a8_RK3588.rkllm	W8A8

Repository Structure

.
├── RK3576
│   ├── Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm
│   └── Qwen2.5-1.5B-Instruct_w8a8_RK3576.rkllm
└── RK3588
    └── Qwen2.5-1.5B-Instruct_w8a8_RK3588.rkllm

Usage

Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching RK3576 or RK3588 platform.

For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.

Docker Deployment

Use the model-specific image for the fastest deployment. The model is included in the image, so no separate model download is required:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  ghcr.io/hanzo-huang/rkllm-docker/qwen2.5-1.5b-instruct:w4a16-rk3576

Alternatively, use the reusable RKLLM environment image and mount a downloaded model:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -v ./:/app/models:ro \
  -e MODEL_PATH=/app/models/RK3576/Qwen2.5-1.5B-Instruct_w4a16_RK3576.rkllm \
  -e TARGET_PLATFORM=rk3576 \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest

The second command requires downloading the selected .rkllm file first and setting MODEL_PATH to its mounted path. Use the matching image, model variant, and TARGET_PLATFORM for the target Rockchip SoC.

Acknowledgements

This repository contains converted RKLLM versions of Qwen2.5-1.5B-Instruct.

Thanks to the Alibaba Cloud Qwen Team for releasing the original model.