Qwen2.5-Coder-7B-Instruct — RK3576 (RKLLM)
Conversion of Qwen/Qwen2.5-Coder-7B-Instruct to RKLLM format for RK3576 NPU (NanoPi R76S, Orange Pi RK3576, and similar boards).
Conversion details
- Toolkit: rkllm-toolkit 1.2.3
- Compatible runtime: rkllm-runtime >= 1.2.3
- Quantization: w4a16
- NPU cores: 2 (dual-core)
- Max context: 4096 tokens
- Platform: RK3576 (NOT compatible with RK3588)
Usage with RKLLama
Place the .rkllm file in your models folder and create a Modelfile with the following content:
FROM="Qwen2.5-Coder-7B-Instruct-rk3576.rkllm" HUGGINGFACE_PATH="Qwen/Qwen2.5-Coder-7B-Instruct" TOKENIZER="/models/tokenizers/Qwen2.5-Coder-7B"
Performance (NanoPi R76S, 16GB RAM)
- ~10-15 tok/s on dual-core NPU
Important note
This model was compiled specifically for RK3576. If you're using an RK3588 (RK3588S, RK3588J), look for a different conversion — they are not binary compatible with each other.
- Downloads last month
- 15
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js