README.md

---
library_name: transformers
license: apache-2.0
base_model:
  - openbmb/MiniCPM-V-4.6
pipeline_tag: image-text-to-text
tags:
  - minicpm-v
  - vlm
  - video-understanding
  - axera
  - ax650
language:
  - zh
  - en
---

# MiniCPM-V-4.6 on AXERA NPU

Ready-to-run deployment package for `openbmb/MiniCPM-V-4.6` on AX650 / NPU3.

- This release packages the AX650 `axllm` runtime together with the compiled text and vision `.axmodel` files.
- The packaged text runtime uses the non-GPTQ BF16 build.
- The packaged vision runtime uses a fixed-shape `448x448` MiniCPM-V-4.6 vision encoder.
- The package supports text-only chat, single-image understanding, and video understanding through the OpenAI-compatible `axllm serve` API.
- The package also includes board-side and server-side Python reference scripts for reference use and comparison.

## Supported Platform

- [x] AX650 / NPU3

## Validated Devices

This package has been validated on the following AX650-based device:

- AX650 / NPU3 development board

## Performance

All measurements below were taken on AX650 / NPU3. `TTFT` stands for time to first token. In this table, `TTFT` is measured end-to-end from request arrival at `axllm serve` to the first generated token, so the multimodal rows include media preprocessing and vision encoding time.

The text-only smoke prompt was kept within one `128`-token prefill chunk. To avoid one-time startup effects, the text row below excludes the first request after service startup. Its `Decode` figure was measured with longer text-only generations (`max_tokens=256`) to better reflect sustained decode throughput; very short smoke replies under-report decode speed because EOS and response-tail overhead become relatively larger. The image row was measured with the packaged fixed-shape `448x448` vision encoder and `assets/sample.png`. The video row used the packaged sample video with `video:assets/red-panda-openai.mp4:2`.

| Scenario | Input tokens | Prefill chunks | TTFT | Decode |
|---|---:|---:|---:|---:|
| Text-only smoke prompt | `25` | `1 x 128` | `275.88 ms avg` (`274.97-276.78 ms`) | `19.12 token/s avg` |
| Image prompt | `88` | `1 x 128` | `729.89 ms avg` (`723.81-741.89 ms`) | `19.02 token/s avg` |
| Video prompt | `1271` | `10 x 128` | `9652.87 ms avg` (`9585.79-9735.26 ms`) | `18.84 token/s avg` |

The packaged runtime uses the following context layout:

- `prefill_len=128`
- `kv_cache_len=2047`
- `prefill_max_token_num=1280`

`Input tokens` in the table above refers to the full request length after chat templating, not just the visual soft tokens. For the shipped `448x448` vision encoder, each selected image block contributes `64` visual soft tokens. Under the current packaged runtime settings, the sample video request in this README uses `1271` total input tokens and spans `10` prefill chunks.

### Startup Runtime Footprint

| Item | Value |
|---|---:|
| `Flash total (text + post + vision axmodels)` | `1.42 GiB` (`1458.81 MiB`) |
| `Package flash total (excluding vision_cache/)` | `1.93 GiB` (`1979.79 MiB`) |
| `Runtime CMM increment during board-side startup` | `1.53 GiB` (`1564.55 MiB`) |

The runtime CMM value above was measured during board-side startup on the validated AX650 board configuration and should be treated as a practical reference value.

## Vision Encoder Latency

Measured on AX650 / NPU3 with `/opt/bin/ax_run_model -m minicpmv4_6_vision_448.axmodel -g 0 -w 1 -r 5`.

| Model | Resolution | Soft Tokens | Time (ms) |
|---|---|---|---:|
| `minicpmv4_6_vision_448.axmodel` | `448x448` | `64` | `234.827 ms avg` |

For this packaged AX650 runtime, the visual token count is fixed by the shipped vision encoder configuration:

- `vision_width = 448`
- `vision_height = 448`
- `vision_patch_size = 14`
- patch grid = `(448 / 14) x (448 / 14) = 32 x 32`
- raw patch tokens = `32 x 32 = 1024`
- current packaged build uses the `16x` visual compression path
- `Soft Tokens = 1024 / 16 = 64`

So, for the fixed-shape runtime shipped in this repository, the relation is:

```text
Soft Tokens = (vision_width / patch_size) x (vision_height / patch_size) / 16
```

`Input tokens` in the performance table can be larger than the visual `Soft Tokens` because `axllm` counts the full templated request, including user text and chat-template tokens in addition to the visual tokens. For the packaged `assets/sample.png` request in this README, the runtime reports `input_num_token=88`, which still fits within a single `128`-token prefill chunk.

`Soft Tokens` is not a runtime-configurable value in this package. This repository ships only `minicpmv4_6_vision_448.axmodel`, so the board-side AX650 runtime always uses `448x448 -> 64` soft tokens for image encoding.

## Package Layout

```text
.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   └── sample.png
├── python/
│   ├── infer_axmodel.py
│   ├── infer_torch.py
│   └── minicpm_v46_tokenizer/
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p128_l0_together.axmodel
├── ...
├── qwen3_5_text_p128_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt
```

This package uses a hybrid layout: the packaged `axllm` runtime plus the compiled `.axmodel` files live at the repository root, while the Python reference scripts and the tokenizer directory used by those scripts stay under `python/`.

## Sample Image

Both the `axllm` flow and the packaged Python examples can use the sample image:
`assets/sample.png`

![sample](assets/sample.png)

## Sample Video

The package also includes a packaged sample video for board-side video understanding validation:

- `assets/red-panda-openai.mp4`

## Direct Inference with `axllm`

> The `axllm` workflow is still being refined. The instructions below reflect the current validated flow and may be adjusted as the packaging continues to evolve.

### Download the Model Package

Download the release package from Hugging Face:

```shell
mkdir -p AXERA-TECH/MiniCPM-V-4.6
cd AXERA-TECH/MiniCPM-V-4.6
hf download AXERA-TECH/MiniCPM-V-4.6 --local-dir .
```

### Install `axllm`

Option 1: use the validated binary included in this repository:

```bash
chmod +x ./bin/axllm
```

Option 2: install `axllm` from the public repository:

```shell
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh
```

Option 3: install with a one-line command:

```shell
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash
```

Option 4: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated `axllm` binary from GitHub Actions:
`https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm`
Then run:

```shell
chmod +x axllm
sudo mv axllm /usr/bin/axllm
```

### Run on the Board

The package root is already arranged for `axllm`, so no extra runtime path arguments are required.

For multimodal testing, you can use the packaged sample image shown above: `./assets/sample.png`, or the packaged sample video: `./assets/red-panda-openai.mp4`.

```bash
./bin/axllm run .
```

In interactive mode:

- press `Enter` directly for text-only chat
- input an image path for single-image chat
- input `video:/path/to/frames_dir` or `video:/path/to/video.mp4` for video chat

### Serve with `axllm`

From the package root on the board:

```bash
./bin/axllm serve . --port 8000
```

Expected model id:

```text
AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047
```

Health check:

```bash
curl http://127.0.0.1:8000/health
```

A typical startup log looks like this:

```text
INF Init | LLM init start
INF Init | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
INF Init | attention config: layers=24 sliding=0 full=6 linear=18 sliding_window=0 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
...
INF Init | max_token_len : 2047
INF Init | kv_cache_size : 512, kv_cache_num: 2047
INF init_groups_from_model | prefill_token_num : 128
INF init_groups_from_model | prefill_max_token_num : 1280
INF Init | MiniCPM-V-4.6 token ids: image_pad=248056 video_pad=248057
INF Init | VisionModule init ok: type=MiniCPMV46VL, tokens_per_block=64, embed_size=1024, out_dtype=fp32
INF Init | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047
```

You can then send requests to the server using the API endpoints shown in the log. For example, to check the health status and list the available models:

```bash
curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

Example output:

```json
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}
{
  "data": [
    {
      "created": 1780908633,
      "id": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}
```

![openai_api_demo](assets/openai_api_demo.png)

### Text Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is 1+1? Reply with the number only."}
        ]
      }
    ],
    "max_tokens": 32
  }'
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "1+1 is 2."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Image Request

```bash
python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please briefly describe this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 64,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=60) as resp:
    print(resp.read().decode())
PY
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "好的，这张图片是一个风格化的红色龙虾卡通形象。它有着夸张的表情和动态的姿势，显得非常活泼和有力。龙虾的肢体姿态显示出它正在准备出击或展示它的力量，整体设计充满了动感和趣味性。这个形象可能用于装饰或象征某种活力和力量。"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Video Request

`axllm serve` accepts either a frames directory or a raw video file:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "video:/path/to/frames_dir"}},
          {"type": "text", "text": "Describe this video."}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

For a raw video file, use `video:/path/to/video.mp4`. If you need to request a specific sampling FPS, use the form `video:/path/to/video.mp4:2`.

To test the packaged sample video from the package root, you can set:

```bash
VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"
```

and then use `video:${VIDEO_PATH}:2` in the request payload.

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "好的，这是当前对视频内容的详细描述：画面中，两只红熊猫正在一个由竹竿搭建的攀爬架周围活动。一只红熊猫正趴在竹竿上，身体伸展，尾巴自然垂落；另一只红熊猫则蹲在下方，抬头向上，似乎正在尝试攀爬或探索竹竿结构。背景是绿色的围栏和草地，整个场景展现了它们活泼、好奇的互动状态。"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Browser UI with `lite_webui`

If you want a browser UI for the OpenAI-compatible service started by `axllm serve`, use [AXERA-TECH/lite_webui](https://huggingface.co/AXERA-TECH/lite_webui/tree/main).

Set the OpenAI base URL to `http://<board-ip>:8000` and the model name to `AXERA-TECH/MiniCPM-V-4.6-AX650-C128-P1152-CTX2047`.

## Python Runtime Requirements

Install the following packages before using the packaged Python reference scripts:

- Board-side `infer_axmodel.py`:
  `pyaxengine`, `transformers`, `numpy`, `ml_dtypes`
- Server-side `infer_torch.py`:
  `torch`, `transformers`

The packaged Python scripts are reference utilities rather than the main runtime path.

## Legacy Python Demo Flow

### Text-Only Inference

`python/infer_axmodel.py` is intended for board-side text debugging of the packaged runtime files:

```bash
cd python
python3 infer_axmodel.py \
  --hf-model ./minicpm_v46_tokenizer \
  --axmodel-dir .. \
  --mode generate \
  --prompt "What is 1+1? Reply with the number only." \
  --prompt-mode prefill \
  --max-new-tokens 16 \
  --kv-cache-len 2047
```

### Hugging Face Reference Inference

`python/infer_torch.py` is intended for x86 or GPU-side comparison against the original Hugging Face model:

```bash
cd python
python infer_torch.py \
  --model-path /path/to/original/MiniCPM-V-4.6 \
  --prompt "Please give a short self introduction."
```

## Packaged Python Runtime Paths

The packaged Python helper paths are:

- `python/infer_axmodel.py`
- `python/infer_torch.py`
- `python/minicpm_v46_tokenizer/`

The packaged `axllm` runtime does not depend on `python/minicpm_v46_tokenizer/`, but `python/infer_axmodel.py` uses it by default.

These path arguments apply to the Python demo flow only. The `axllm` flow reads the same root-level runtime files packaged in this repository.

## Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

- Original Hugging Face model: [openbmb/MiniCPM-V-4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6)
- AXERA conversion and deployment workflow: [AXERA-TECH/MiniCPM-V-4.6.axera](https://github.com/AXERA-TECH/MiniCPM-V-4.6.axera)

## Discussion

- GitHub Issues
- QQ group: `139953715`