license: gemma
base_model: google/gemma-3-4b-it
library_name: rkllm
pipeline_tag: text-generation
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- gemma
- gemma-3
gemma-3-4b-it-RKLLM
RKLLM-converted versions of Gemma 3 4B IT for deployment on Rockchip RK3576 and RK3588 NPUs.
.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.
Base Model
- Model: Gemma 3 4B IT
- Author: Google
- Original Model: google/gemma-3-4b-it
- Original License: Gemma
Refer to the upstream model card for the original model’s capabilities, terms, and limitations.
Supported Platforms
| SoC | Quantization | Status |
|---|---|---|
| RK3576 | W4A16 | ✅ Tested |
| RK3576 | W8A8 | ✅ Tested |
| RK3588 | W8A8 | ✅ Supported |
RK3576 W4A16 and W8A8 are tested. RK3588 W8A8 is supported as a target platform but is not marked as tested.
Conversion
- RKLLM Toolkit: 1.2.3
- Target Platforms: RK3576, RK3588
Available Variants
| File | Quantization |
|---|---|
| gemma-3-4b-it_w4a16_RK3576.rkllm | W4A16 |
| gemma-3-4b-it_w8a8_RK3576.rkllm | W8A8 |
| gemma-3-4b-it_w8a8_RK3588.rkllm | W8A8 |
Repository Structure
.
├── gemma-3-4b-it_data_quant.json
├── RK3576
│ ├── gemma-3-4b-it_w4a16_RK3576.rkllm
│ └── gemma-3-4b-it_w8a8_RK3576.rkllm
└── RK3588
└── gemma-3-4b-it_w8a8_RK3588.rkllm
Usage
Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching Rockchip platform.
For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.
Docker Deployment
Use the model-specific image for the fastest RK3576 W4A16 deployment. The model is included in the image:
sudo docker run --rm -it \
--privileged \
-p 8001:8001 \
-v /dev:/dev \
ghcr.io/hanzo-huang/rkllm-docker/gemma-3-4b-it:w4a16-rk3576
Alternatively, use the reusable environment image and mount a downloaded model:
sudo docker run --rm -it \
--privileged \
-p 8001:8001 \
-v /dev:/dev \
-v ./:/app/models:ro \
-e MODEL_PATH=/app/models/RK3576/gemma-3-4b-it_w4a16_RK3576.rkllm \
-e TARGET_PLATFORM=rk3576 \
ghcr.io/hanzo-huang/rkllm-docker:env-latest
The second command requires downloading the selected .rkllm file first. Use the matching model variant and target platform.
License
The original Gemma 3 4B IT model is released under the Gemma license. Converted artifacts are derived from the original model and remain subject to the applicable upstream license terms.
Acknowledgements
Thanks to Google for releasing the original Gemma model and to Rockchip and RKLLM contributors for the deployment toolchain.