| # Qwen2 with 1.5B ONNX INT8 — Mobile-Optimized Quantized Model |
|
|
| ## 📌 Description |
| This is a quantized INT8 version of the Qwen language model, |
| converted to ONNX format for efficient inference on low-resource |
| devices such as mobile phones and edge hardware. |
|
|
| ## 🎯 Objective |
| The goal of this project is to make Qwen accessible on devices |
| with limited memory and compute power (e.g. iPhone 12, mid-range |
| Android phones) without requiring high-end GPUs or cloud infrastructure. |
|
|
| ## ⚡ Key Features |
| - **Format**: ONNX (INT8 quantized) |
| - **Size**: Reduced ~4x compared to FP32 original |
| - **Target hardware**: CPU-only devices, mobile, edge |
| - **Inference engine**: ONNX Runtime (`onnxruntime`) |
|
|
| ## 🛠️ Usage |
|
|
| ### Install dependencies |
| pip install onnxruntime numpy |
|
|
| ### Run inference |
| import onnxruntime as ort |
|
|
| session = ort.InferenceSession( |
| "model_int8.onnx", |
| providers=["CPUExecutionProvider"] |
| ) |
| |
| ## 📁 Files |
| | File | Description | |
| |---|---| |
| | `model.onnx` | Original FP32 model | |
| | `model_int8.onnx` | INT8 quantized model ✅ | |
| | `tokenizer.json` | Tokenizer config | |
| | `vocab.json` | Vocabulary | |
| | `quantize_int8.py` | Quantization script | |
|
|
|  |
|
|
