File size: 2,105 Bytes
cc321bf d2a74b1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
license: apache-2.0
language:
- en
- zh
base_model:
- openbmb/MiniCPM-V-4
pipeline_tag: image-text-to-text
library_name: litert
tags:
- minicpm
- minicpm-v-4
- litert
- tflite
- on-device
- edge-ai
---
# MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU)
`MiniCPM-V-4-int8.litertlm` — an on-device [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) build of MiniCPM-V-4, quantized to **INT8** (weight-only, GPTQ + Hadamard rotation) and packaged as a single `.litertlm` bundle for CPU inference on **Snapdragon 8850**.
## Model
For the base model, architecture, capabilities and license, refer to the original model card:
**https://huggingface.co/openbmb/MiniCPM-V-4**
This repository only provides an **edge-optimized LiteRT deployment** of that model:
- **Format**: `.litertlm` (LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler).
- **Quantization**: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing).
- **Vision**: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP.
## Performance (Snapdragon 8850, CPU backend)
Measured on-device (arm64 CPU, `--backend=cpu`), single image, 4-slice input:
| Metric | Value |
|---|---|
| Prefill speed | **40.9 tokens/sec** |
| Decode speed | **16.6 tokens/sec** |
| Vision encode (4 slices) | **~2.04 s** |
| Time to first token | ~7.0 s (284-token prefill incl. vision) |
| Init (model load + compile) | ~2.2 s |
## Accuracy — MME benchmark (on-device, 8850 CPU)
Full MME evaluation running the INT8 bundle on-device:
| Group | Score |
|---|---|
| **Perception** | **1568** |
| **Cognition** | **491** |
| **Total** | **2059** |
## Usage
Run with a LiteRT-LM CPU runner:
```bash
LD_LIBRARY_PATH=. ./litert_lm_main \
--model_path=./MiniCPM-V-4-int8.litertlm \
--backend=cpu \
--image_path=./image.jpg \
--input_prompt="What is in this image?"
```
## License
Follows the license of the base model — see **https://huggingface.co/openbmb/MiniCPM-V-4**.
|