|
|
---
|
|
|
license: mit
|
|
|
base_model: microsoft/Phi-3-mini-4k-instruct
|
|
|
tags:
|
|
|
- rkllm
|
|
|
- rk3588
|
|
|
- npu
|
|
|
- phi3
|
|
|
- quantized
|
|
|
- w8a8
|
|
|
language:
|
|
|
- en
|
|
|
pipeline_tag: text-generation
|
|
|
---
|
|
|
|
|
|
# Phi-3-mini-4k-instruct — RKLLM (w8a8) for RK3588
|
|
|
|
|
|
Pre-quantized **w8a8** build of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) ready to run on Rockchip **RK3588 / RK3588S** NPU via the [RKLLM runtime v1.2.3](https://github.com/airockchip/rknn-llm).
|
|
|
|
|
|
| Detail | Value |
|
|
|
|---|---|
|
|
|
| Parameters | 3.82 B |
|
|
|
| Quantization | w8a8 (8-bit weights, 8-bit activations) |
|
|
|
| Context length | 4 096 tokens |
|
|
|
| NPU cores | 3 |
|
|
|
| File size | ~3.7 GB |
|
|
|
| Inference speed | **6.82 tokens/sec** (measured on RK3588, 3 cores) |
|
|
|
| Prefill speed | 49.22 tokens/sec |
|
|
|
| RAM usage | ~3.7 GB |
|
|
|
| Toolkit version | rkllm-toolkit 1.2.3 |
|
|
|
| Runtime version | rkllm runtime 1.2.3 |
|
|
|
| Calibration | 20 diverse prompts (general knowledge, code, math, science) |
|
|
|
|
|
|
## What is Phi-3?
|
|
|
|
|
|
Phi-3-Mini-4K-Instruct is a 3.8B parameter lightweight model from Microsoft, trained on 4.9 trillion tokens. It excels at reasoning, math, and code generation while being small enough for edge deployments. MIT licensed.
|
|
|
|
|
|
## How to use
|
|
|
|
|
|
### 1. Download
|
|
|
|
|
|
Place the `.rkllm` file and `config.json` in a model directory on your RK3588 board:
|
|
|
|
|
|
```bash
|
|
|
mkdir -p ~/models/Phi-3-mini-4k-instruct
|
|
|
# Copy both files into this directory
|
|
|
```
|
|
|
|
|
|
### 2. Run with rkllm binary
|
|
|
|
|
|
```bash
|
|
|
cd ~/models/Phi-3-mini-4k-instruct
|
|
|
rkllm Phi-3-mini-4k-instruct-w8a8.rkllm
|
|
|
```
|
|
|
|
|
|
The RKLLM runtime applies Phi-3's chat template (`<|system|>...<|end|><|user|>...<|end|><|assistant|>`) internally at the token level. Send plain text only.
|
|
|
|
|
|
### 3. Chat template
|
|
|
|
|
|
Phi-3 uses a distinctive chat format:
|
|
|
```
|
|
|
<|system|>
|
|
|
You are a helpful assistant.<|end|>
|
|
|
<|user|>
|
|
|
How does photosynthesis work?<|end|>
|
|
|
<|assistant|>
|
|
|
```
|
|
|
|
|
|
The runtime handles this automatically — no manual template needed.
|
|
|
|
|
|
## Conversion details
|
|
|
|
|
|
**Toolkit:** rkllm-toolkit 1.2.3 (Python 3.10, x86_64)
|
|
|
|
|
|
```python
|
|
|
from rkllm.api import RKLLM
|
|
|
|
|
|
llm = RKLLM()
|
|
|
llm.load_huggingface(model="./model", model_lora=None, device="cpu")
|
|
|
llm.build(
|
|
|
do_quantization=True,
|
|
|
optimization_level=1,
|
|
|
quantized_dtype="w8a8",
|
|
|
quantized_algorithm="normal",
|
|
|
target_platform="rk3588",
|
|
|
num_npu_core=3,
|
|
|
extra_qparams=None,
|
|
|
dataset="./data_quant.json",
|
|
|
max_context=4096,
|
|
|
)
|
|
|
llm.export_rkllm("./Phi-3-mini-4k-instruct-w8a8.rkllm")
|
|
|
```
|
|
|
|
|
|
**Calibration dataset:** 20 prompts covering general knowledge, code, math, science, and creative writing, formatted with Phi-3's native chat template via `tokenizer.apply_chat_template()`.
|
|
|
|
|
|
## Notes
|
|
|
|
|
|
- **No `<think>` tags:** Phi-3 is not a reasoning model — it does not produce chain-of-thought in `<think>...</think>` tags. For thinking mode, use Qwen3 models.
|
|
|
- **Strong at math/code:** Despite its size, Phi-3 scores 85.7% on GSM8K and 57.3% on HumanEval.
|
|
|
- **English-primary:** Best performance in English. Other languages have reduced quality.
|
|
|
|
|
|
## Hardware tested
|
|
|
|
|
|
- Orange Pi 5 Plus (RK3588, 16 GB RAM)
|
|
|
- RKNPU driver 0.9.8
|
|
|
- RKLLM runtime 1.2.3
|
|
|
|
|
|
## License
|
|
|
|
|
|
MIT — same as the base model.
|
|
|
|
