Qwen3-4B-RKLLM

RKLLM-converted versions of Qwen3-4B for deployment on Rockchip RK3576 and RK3588 NPUs.

.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.

Base Model

• Model: Qwen3-4B
• Author: Alibaba Cloud Qwen Team
• Original Model: Qwen/Qwen3-4B
• Original License: Apache-2.0

Refer to the upstream model card for the original model’s capabilities and limitations.

Supported Platforms

SoC	Quantization	Status
RK3576	W4A16	✅ Tested
RK3576	W8A8	✅ Tested
RK3588	W8A8	✅ Supported

RK3576 W4A16 and W8A8 are tested. RK3588 W8A8 is supported as a target platform but is not marked as tested.

Conversion

• RKLLM Toolkit: 1.2.3
• Target Platforms: RK3576, RK3588

Available Variants

File	Quantization
Qwen3-4B_w4a16_RK3576.rkllm	W4A16
Qwen3-4B_w8a8_RK3576.rkllm	W8A8
Qwen3-4B_w8a8_RK3588.rkllm	W8A8

Repository Structure

.
├── Qwen3-4B_data_quant.json
├── RK3576
│   ├── Qwen3-4B_w4a16_RK3576.rkllm
│   └── Qwen3-4B_w8a8_RK3576.rkllm
└── RK3588
    └── Qwen3-4B_w8a8_RK3588.rkllm

Usage

Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching Rockchip platform.

For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.

Docker Deployment

Use the model-specific image for the fastest RK3576 W4A16 deployment. The model is included in the image:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  ghcr.io/hanzo-huang/rkllm-docker/qwen3-4b:w4a16-rk3576

Alternatively, use the reusable environment image and mount a downloaded model:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -v ./:/app/models:ro \
  -e MODEL_PATH=/app/models/RK3576/Qwen3-4B_w4a16_RK3576.rkllm \
  -e TARGET_PLATFORM=rk3576 \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest

The second command requires downloading the selected .rkllm file first. Use the matching model variant and target platform.

License

The original Qwen3-4B model is released under the Apache-2.0 license. Converted artifacts are derived from the original model and remain subject to the applicable upstream license terms.

Acknowledgements

Thanks to the Alibaba Cloud Qwen Team for releasing the original model and to Rockchip and RKLLM contributors for the deployment toolchain.