metadata
license: apache-2.0
language:
- en
- zh
base_model:
- openbmb/MiniCPM-V-4
pipeline_tag: image-text-to-text
library_name: litert
tags:
- minicpm
- minicpm-v-4
- litert
- tflite
- on-device
- edge-ai
MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU)
MiniCPM-V-4-int8.litertlm — an on-device LiteRT-LM build of MiniCPM-V-4, quantized to INT8 (weight-only, GPTQ + Hadamard rotation) and packaged as a single .litertlm bundle for CPU inference on Snapdragon 8850.
Model
For the base model, architecture, capabilities and license, refer to the original model card:
https://huggingface.co/openbmb/MiniCPM-V-4
This repository only provides an edge-optimized LiteRT deployment of that model:
- Format:
.litertlm(LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler). - Quantization: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing).
- Vision: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP.
Performance (Snapdragon 8850, CPU backend)
Measured on-device (arm64 CPU, --backend=cpu), single image, 4-slice input:
| Metric | Value |
|---|---|
| Prefill speed | 40.9 tokens/sec |
| Decode speed | 16.6 tokens/sec |
| Vision encode (4 slices) | ~2.04 s |
| Time to first token | ~7.0 s (284-token prefill incl. vision) |
| Init (model load + compile) | ~2.2 s |
Accuracy — MME benchmark (on-device, 8850 CPU)
Full MME evaluation running the INT8 bundle on-device:
| Group | Score |
|---|---|
| Perception | 1568 |
| Cognition | 491 |
| Total | 2059 |
Usage
Run with a LiteRT-LM CPU runner:
LD_LIBRARY_PATH=. ./litert_lm_main \
--model_path=./MiniCPM-V-4-int8.litertlm \
--backend=cpu \
--image_path=./image.jpg \
--input_prompt="What is in this image?"
License
Follows the license of the base model — see https://huggingface.co/openbmb/MiniCPM-V-4.