| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| base_model: |
| - openbmb/MiniCPM-V-4 |
| pipeline_tag: image-text-to-text |
| library_name: litert |
| tags: |
| - minicpm |
| - minicpm-v-4 |
| - litert |
| - tflite |
| - on-device |
| - edge-ai |
| --- |
| |
| # MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU) |
|
|
| `MiniCPM-V-4-int8.litertlm` — an on-device [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) build of MiniCPM-V-4, quantized to **INT8** (weight-only, GPTQ + Hadamard rotation) and packaged as a single `.litertlm` bundle for CPU inference on **Snapdragon 8850**. |
|
|
| ## Model |
|
|
| For the base model, architecture, capabilities and license, refer to the original model card: |
|
|
| **https://huggingface.co/openbmb/MiniCPM-V-4** |
|
|
| This repository only provides an **edge-optimized LiteRT deployment** of that model: |
|
|
| - **Format**: `.litertlm` (LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler). |
| - **Quantization**: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing). |
| - **Vision**: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP. |
|
|
| ## Performance (Snapdragon 8850, CPU backend) |
|
|
| Measured on-device (arm64 CPU, `--backend=cpu`), single image, 4-slice input: |
|
|
| | Metric | Value | |
| |---|---| |
| | Prefill speed | **40.9 tokens/sec** | |
| | Decode speed | **16.6 tokens/sec** | |
| | Vision encode (4 slices) | **~2.04 s** | |
| | Time to first token | ~7.0 s (284-token prefill incl. vision) | |
| | Init (model load + compile) | ~2.2 s | |
|
|
| ## Accuracy — MME benchmark (on-device, 8850 CPU) |
|
|
| Full MME evaluation running the INT8 bundle on-device: |
|
|
| | Group | Score | |
| |---|---| |
| | **Perception** | **1568** | |
| | **Cognition** | **491** | |
| | **Total** | **2059** | |
|
|
| ## Usage |
|
|
| Run with a LiteRT-LM CPU runner: |
|
|
| ```bash |
| LD_LIBRARY_PATH=. ./litert_lm_main \ |
| --model_path=./MiniCPM-V-4-int8.litertlm \ |
| --backend=cpu \ |
| --image_path=./image.jpg \ |
| --input_prompt="What is in this image?" |
| ``` |
|
|
| ## License |
|
|
| Follows the license of the base model — see **https://huggingface.co/openbmb/MiniCPM-V-4**. |
|
|
