| license: apache-2.0 | |
| base_model: Qwen/Qwen2.5-Coder-7B-Instruct | |
| tags: | |
| - rkllm | |
| - rk3576 | |
| - npu | |
| - qwen2.5-coder | |
| - edge-ai | |
| # Qwen2.5-Coder-7B-Instruct — RK3576 (RKLLM) | |
| Conversion of [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) to RKLLM format for **RK3576** NPU (NanoPi R76S, Orange Pi RK3576, and similar boards). | |
| ## Conversion details | |
| - **Toolkit:** rkllm-toolkit 1.2.3 | |
| - **Compatible runtime:** rkllm-runtime >= 1.2.3 | |
| - **Quantization:** w4a16 | |
| - **NPU cores:** 2 (dual-core) | |
| - **Max context:** 4096 tokens | |
| - **Platform:** RK3576 (NOT compatible with RK3588) | |
| ## Usage with RKLLama | |
| Place the `.rkllm` file in your models folder and create a `Modelfile` with the following content: | |
| FROM="Qwen2.5-Coder-7B-Instruct-rk3576.rkllm" | |
| HUGGINGFACE_PATH="Qwen/Qwen2.5-Coder-7B-Instruct" | |
| TOKENIZER="/models/tokenizers/Qwen2.5-Coder-7B" | |
| ## Performance (NanoPi R76S, 16GB RAM) | |
| - ~10-15 tok/s on dual-core NPU | |
| ## Important note | |
| This model was compiled specifically for RK3576. If you're using an RK3588 (RK3588S, RK3588J), look for a different conversion — they are not binary compatible with each other. | |