README.md

---

language:
  - en
  - zh
license: apache-2.0
library_name: rkllm
tags:
  - rkllm
  - rknn
  - rk3588
  - npu
  - qwen3-vl
  - vision-language
  - orange-pi
  - edge-ai
  - ocr
base_model: Qwen/Qwen3-VL-2B-Instruct
pipeline_tag: image-text-to-text
---


# Qwen3-VL-2B-Instruct for RKLLM v1.2.3 (RK3588 NPU)

Pre-converted [Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) for the **Rockchip RK3588 NPU** using [rknn-llm](https://github.com/airockchip/rknn-llm) runtime v1.2.3.

Runs on **Orange Pi 5 Plus**, **Rock 5B**, **Radxa NX5**, and other RK3588-based SBCs with 8GB+ RAM.

## Files

| File | Size | Description |
|---|---|---|
| `qwen3-vl-2b-instruct_w8a8_rk3588.rkllm` | 2.3 GB | LLM decoder (W8A8 quantized) — shared by all vision resolutions |
| `qwen3-vl-2b_vision_448_rk3588.rknn` | 812 MB | Vision encoder @ 448×448 (default, 196 tokens) |
| `qwen3-vl-2b_vision_672_rk3588.rknn` | 854 MB | Vision encoder @ 672×672 (441 tokens) ⭐ **Recommended** |
| `qwen3-vl-2b_vision_896_rk3588.rknn` | 923 MB | Vision encoder @ 896×896 (784 tokens) |

## Choosing a Vision Encoder Resolution

The LLM decoder (`.rkllm`) is resolution-independent — only the vision encoder (`.rknn`) changes. Place **one** `.rknn` file alongside the `.rkllm` in your model directory, or rename alternatives to `.rknn.alt` to disable them.

| Resolution | Visual Tokens | Encoder Time* | Total Response* | Best For |
|---|---|---|---|---|
| **448×448** | 196 (14×14) | ~2s | ~5-10s | General scene description, fast responses |
| **672×672** ⭐ | 441 (21×21) | ~4s | ~9-11s | **Balanced: good detail + reasonable speed** |
| **896×896** | 784 (28×28) | ~12s | ~25-28s | Maximum detail, fine text/OCR tasks |

\*Measured on Orange Pi 5 Plus (16GB) with 14MB JPEG input, single image.



### Resolution Math



Qwen3-VL uses `patch_size=16` and `merge_size=2`, so:

- Resolution must be **divisible by 32** (16 × 2)

- Visual tokens = (height/32)² = 196 / 441 / 784 for 448 / 672 / 896



Higher resolution = more visual tokens = better fine detail but:

- Proportionally more NPU compute for the vision encoder

- More tokens for the LLM to process (longer prefill)

- Same decode speed (~15 tok/s) — only "time to first token" increases



## Quick Start



### Directory Structure



```

~/models/Qwen3-VL-2b/

    qwen3-vl-2b-instruct_w8a8_rk3588.rkllm   # LLM decoder (always needed)

    qwen3-vl-2b_vision_672_rk3588.rknn        # Active vision encoder

    qwen3-vl-2b_vision_448_rk3588.rknn.alt    # Alternative (inactive)

    qwen3-vl-2b_vision_896_rk3588.rknn.alt    # Alternative (inactive)

```



### Switching Resolution



To switch to a different resolution, rename the files:



```bash

cd ~/models/Qwen3-VL-2b/



# Deactivate current encoder

mv qwen3-vl-2b_vision_672_rk3588.rknn qwen3-vl-2b_vision_672_rk3588.rknn.alt



# Activate the 896 encoder

mv qwen3-vl-2b_vision_896_rk3588.rknn.alt qwen3-vl-2b_vision_896_rk3588.rknn



# Restart your API server

sudo systemctl restart rkllm-api

```



### Using with RKLLM API Server



This model is designed for use with the [RKLLM API Server](https://github.com/jdacostap/rkllm-api), which provides an OpenAI-compatible API for RK3588 NPU inference. The server auto-detects the vision encoder resolution from the `.rknn` file's input tensor attributes.



## Export Details



### LLM Decoder



- **Source**: [Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)

- **Quantization**: W8A8 (8-bit weights, 8-bit activations)

- **Tool**: rkllm-toolkit v1.2.3

- **Context**: 4096 tokens



### Vision Encoders



- **Source**: Qwen3-VL-2B-Instruct visual encoder weights

- **Export pipeline**: HuggingFace model → ONNX (`export_vision.py`) → RKNN (`export_vision_rknn.py`)

- **Tool**: rknn-toolkit2 v2.3.2

- **Precision**: FP32 (no quantization — vision encoder quality is critical)

- **Target**: rk3588



The 448 encoder was converted with default settings from rknn-llm. The 672 and 896 encoders were re-exported with custom `--height` and `--width` flags to `export_vision.py` and `export_vision_rknn.py` from the [rknn-llm multimodal demo](https://github.com/airockchip/rknn-llm/tree/main/examples/multimodal_model_demo/export).



### Re-exporting at a Custom Resolution



To export the vision encoder at a different resolution (must be divisible by 32):



```bash

# Activate the export environment

source ~/rkllm-env/bin/activate

cd ~/rknn-llm/examples/multimodal_model_demo



# Step 1: Export HuggingFace model to ONNX

python3 export/export_vision.py \

  --path ~/models-hf/Qwen3-VL-2B-Instruct \

  --model_name qwen3-vl \

  --height 672 --width 672 \

  --device cpu



# Step 2: Convert ONNX to RKNN

python3 export/export_vision_rknn.py \

  --path ./onnx/qwen3-vl_vision.onnx \

  --model_name qwen3-vl \

  --target-platform rk3588 \

  --height 672 --width 672

```



**Memory requirements**: ~20 GB RAM (or swap) for 672×672, ~35 GB for 896×896. CPU-only export works fine (no GPU needed).



**Dependencies** (in a Python 3.10 venv):

- `rknn-toolkit2 >= 2.3.2`

- `torch == 2.4.0`

- `transformers >= 4.57.0`

- `onnx >= 1.18.0`



## Performance Benchmarks



Tested on **Orange Pi 5 Plus (16GB RAM)**, RK3588 SoC, RKNPU driver 0.9.8:



| Metric | 448×448 | 672×672 | 896×896 |

|---|---|---|---|

| Vision encode time | ~2 s | ~4 s | ~12 s |

| Total VL response | 5–10 s | 9–11 s | 25–28 s |

| Text-only decode | ~15 tok/s | ~15 tok/s | ~15 tok/s |

| Peak RAM (VL inference) | ~5.5 GB | ~6.5 GB | ~8.5 GB |

| RKNN file size | 812 MB | 854 MB | 923 MB |



## Known Limitations



- **OCR accuracy**: The 2B-parameter LLM is the bottleneck for OCR tasks, not the vision encoder resolution. Higher resolution helps with fine detail but the model may still misread characters.

- **Fixed resolution**: Each `.rknn` file is compiled for a specific input resolution. Images are automatically resized (with aspect-ratio-preserving padding) to match. There is no dynamic resolution switching within a single model file.

- **REGTASK warnings**: The 672 and 896 encoders produce "bit width exceeds limit" register-field warnings during RKNN conversion. These are cosmetic in rknn-toolkit2 v2.3.2 and do not affect runtime inference on the RK3588.



## License



Apache 2.0, inherited from [Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct).



## Credits



- **Model**: [Qwen Team](https://huggingface.co/Qwen) for Qwen3-VL-2B-Instruct

- **Runtime**: [Rockchip / airockchip](https://github.com/airockchip/rknn-llm) for rkllm-toolkit and rknn-toolkit2

- **API Server**: [RKLLM API Server](https://github.com/jdacostap/rkllm-api) — OpenAI-compatible server for RK3588 NPU