gemma-3-4b-it-RKLLM

RKLLM-converted versions of Gemma 3 4B IT for deployment on Rockchip RK3576 and RK3588 NPUs.

.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.

Base Model

• Model: Gemma 3 4B IT
• Author: Google
• Original Model: google/gemma-3-4b-it
• Original License: Gemma

Refer to the upstream model card for the original model’s capabilities, terms, and limitations.

Supported Platforms

SoC	Quantization	Status
RK3576	W4A16	✅ Tested
RK3576	W8A8	✅ Tested
RK3588	W8A8	✅ Supported

RK3576 W4A16 and W8A8 are tested. RK3588 W8A8 is supported as a target platform but is not marked as tested.

Conversion

• RKLLM Toolkit: 1.2.3
• Target Platforms: RK3576, RK3588

Available Variants

File	Quantization
gemma-3-4b-it_w4a16_RK3576.rkllm	W4A16
gemma-3-4b-it_w8a8_RK3576.rkllm	W8A8
gemma-3-4b-it_w8a8_RK3588.rkllm	W8A8

Repository Structure

.
├── gemma-3-4b-it_data_quant.json
├── RK3576
│   ├── gemma-3-4b-it_w4a16_RK3576.rkllm
│   └── gemma-3-4b-it_w8a8_RK3576.rkllm
└── RK3588
    └── gemma-3-4b-it_w8a8_RK3588.rkllm

Usage

Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching Rockchip platform.

For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.

Docker Deployment

Use the model-specific image for the fastest RK3576 W4A16 deployment. The model is included in the image:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  ghcr.io/hanzo-huang/rkllm-docker/gemma-3-4b-it:w4a16-rk3576

Alternatively, use the reusable environment image and mount a downloaded model:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -v ./:/app/models:ro \
  -e MODEL_PATH=/app/models/RK3576/gemma-3-4b-it_w4a16_RK3576.rkllm \
  -e TARGET_PLATFORM=rk3576 \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest

The second command requires downloading the selected .rkllm file first. Use the matching model variant and target platform.

License

The original Gemma 3 4B IT model is released under the Gemma license. Converted artifacts are derived from the original model and remain subject to the applicable upstream license terms.

Acknowledgements

Thanks to Google for releasing the original Gemma model and to Rockchip and RKLLM contributors for the deployment toolchain.