metadata
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
- rkllm
- rk3576
- npu
- qwen2.5-coder
- edge-ai
Qwen2.5-Coder-7B-Instruct — RK3576 (RKLLM)
Conversion of Qwen/Qwen2.5-Coder-7B-Instruct to RKLLM format for RK3576 NPU (NanoPi R76S, Orange Pi RK3576, and similar boards).
Conversion details
- Toolkit: rkllm-toolkit 1.2.3
- Compatible runtime: rkllm-runtime >= 1.2.3
- Quantization: w4a16
- NPU cores: 2 (dual-core)
- Max context: 4096 tokens
- Platform: RK3576 (NOT compatible with RK3588)
Usage with RKLLama
Place the .rkllm file in your models folder and create a Modelfile with the following content:
FROM="Qwen2.5-Coder-7B-Instruct-rk3576.rkllm" HUGGINGFACE_PATH="Qwen/Qwen2.5-Coder-7B-Instruct" TOKENIZER="/models/tokenizers/Qwen2.5-Coder-7B"
Performance (NanoPi R76S, 16GB RAM)
- ~10-15 tok/s on dual-core NPU
Important note
This model was compiled specifically for RK3576. If you're using an RK3588 (RK3588S, RK3588J), look for a different conversion — they are not binary compatible with each other.