README.md

# MiniCPM-V-4 · LiteRT INT8 (on-device, Snapdragon 8850 CPU)

`MiniCPM-V-4-int8.litertlm` — an on-device [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) build of MiniCPM-V-4, quantized to **INT8** (weight-only, GPTQ + Hadamard rotation) and packaged as a single `.litertlm` bundle for CPU inference on **Snapdragon 8850**.

## Model

For the base model, architecture, capabilities and license, refer to the original model card:

**https://huggingface.co/openbmb/MiniCPM-V-4**

This repository only provides an **edge-optimized LiteRT deployment** of that model:

- **Format**: `.litertlm` (LiteRT-LM bundle: tokenizer + LLM prefill/decode + embedder + navit SigLIP vision encoder + resampler).
- **Quantization**: INT8 weight-only, per-channel, GPTQ + Hadamard rotation (activation-outlier smoothing).
- **Vision**: official multi-slice preprocessing (thumbnail + sub-tiles, 64 tokens per slice, up to 9 slices), navit SigLIP.

## Performance (Snapdragon 8850, CPU backend)

Measured on-device (arm64 CPU, `--backend=cpu`), single image, 4-slice input:

| Metric | Value |
|---|---|
| Prefill speed | **40.9 tokens/sec** |
| Decode speed | **16.6 tokens/sec** |
| Vision encode (4 slices) | **~2.04 s** |
| Time to first token | ~7.0 s (284-token prefill incl. vision) |
| Init (model load + compile) | ~2.2 s |

## Accuracy — MME benchmark (on-device, 8850 CPU)

Full MME evaluation running the INT8 bundle on-device:

| Group | Score |
|---|---|
| **Perception** | **1568** |
| **Cognition** | **491** |
| **Total** | **2059** |

## Usage

Run with a LiteRT-LM CPU runner:

```bash
LD_LIBRARY_PATH=. ./litert_lm_main \
  --model_path=./MiniCPM-V-4-int8.litertlm \
  --backend=cpu \
  --image_path=./image.jpg \
  --input_prompt="What is in this image?"
```

## License

Follows the license of the base model — see **https://huggingface.co/openbmb/MiniCPM-V-4**.