File size: 3,503 Bytes
02f0149 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | ---
license: llama3.2
base_model: meta-llama/Llama-3.2-3B-Instruct
tags:
- rkllm
- rk3588
- rockchip
- npu
- quantized
- llama
language:
- en
---
# Llama-3.2-3B-Instruct — RKLLM v1.2.3 (w8a8, RK3588)
RKLLM conversion of [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) for Rockchip RK3588 NPU inference.
Converted with RKLLM Toolkit v1.2.3. This is a standard instruct model — it does **not** produce `<think>` reasoning blocks.
## Key Details
| Property | Value |
|----------|-------|
| Base Model | meta-llama/Llama-3.2-3B-Instruct |
| Toolkit Version | RKLLM Toolkit v1.2.3 |
| Runtime Version | RKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended) |
| Quantization | w8a8 (8-bit weights, 8-bit activations) |
| Quantization Algorithm | normal |
| Target Platform | RK3588 |
| NPU Cores | 3 |
| Max Context Length | 8192 tokens |
| Optimization Level | 0 |
| Thinking Mode | ❌ Not supported |
| Languages | English (+ multilingual inherited from Llama 3.2) |
## Why This Model?
Llama 3.2 3B Instruct is Meta's latest compact instruction model. It brings a different architecture and training lineage to the RK3588 NPU lineup — strong at instruction following, coding, and general reasoning without the overhead of a thinking/reasoning mode.
At ~3B parameters it sits between the 1.7B and 4B Qwen3 models, offering a useful middle ground.
## Hardware Tested
- **Orange Pi 5 Plus** — RK3588, 16GB RAM, Armbian Linux
- RKNPU driver 0.9.8
- RKLLM Runtime v1.2.3
## Usage
### With the official RKLLM API demo
```bash
# Clone the runtime
git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo
# Run (aarch64)
./build/rkllm_api_demo /path/to/Llama-3.2-3B-Instruct-rk3588-w8a8.rkllm 4096 8192
```
### With the RKLLM API Server
Download and place in `~/models/`:
```bash
mkdir -p ~/models/Llama-3.2-3B-Instruct
cd ~/models/Llama-3.2-3B-Instruct
git lfs install && git clone https://huggingface.co/GatekeeperZA/Llama-3.2-3B-Instruct-RKLLM-v1.2.3 .
```
The server auto-detects the model as `instruct` capability (no thinking). Use with [GatekeeperZA/RKLLM-API-Server](https://github.com/GatekeeperZA/RKLLM-API-Server).
## Conversion Script
```python
from rkllm.api import RKLLM
llm = RKLLM()
llm.load_huggingface(model="meta-llama/Llama-3.2-3B-Instruct", device="cpu", dtype="float32")
llm.build(
do_quantization=True,
optimization_level=0,
quantized_dtype="w8a8",
quantized_algorithm="normal",
target_platform="rk3588",
num_npu_core=3,
max_context=8192,
)
llm.export_rkllm("./Llama-3.2-3B-Instruct-rk3588-w8a8.rkllm")
```
> **WSL2 note:** Requires ≥16GB WSL2 memory (`memory=16GB` in `~/.wslconfig`). Write the output to a native Linux path (`/home/user/`) first, then copy to `/mnt/` — writing directly to the Windows mount triggers OOM during the export phase.
## File Listing
| File | Description |
|------|-------------|
| `Llama-3.2-3B-Instruct-rk3588-w8a8.rkllm` | Quantized model for RK3588 NPU |
## Compatibility Notes
- Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
- RKNPU driver: ≥ 0.9.6
- SoCs: RK3588 / RK3588S (3 NPU cores). Not compatible with RK3576 (2 cores) without reconversion.
- RAM: ~3.5GB loaded. Runs comfortably on 8GB+ boards.
## Acknowledgements
- Meta / FAIR for the Llama 3.2 base model
- Rockchip / airockchip for the RKLLM toolkit and runtime
- Converted by [GatekeeperZA](https://huggingface.co/GatekeeperZA)
|