README.md

---
library_name: transformers
license: apache-2.0
base_model:
  - openbmb/MiniCPM-V-4.6
pipeline_tag: image-text-to-text
tags:
  - minicpm-v
  - vlm
  - video-understanding
  - axera
  - ax637
language:
  - zh
  - en
---

# MiniCPM-V-4.6 on AXERA NPU

Ready-to-run deployment package for `openbmb/MiniCPM-V-4.6` on AX637 / NPU1.

- This release packages the AX637 `axllm` runtime together with the compiled text and vision `.axmodel` files.
- The packaged text runtime uses the non-GPTQ w8a16 build.
- The packaged vision runtime uses a fixed-shape `448x448` MiniCPM-V-4.6 vision encoder.
- The package supports text-only chat, single-image understanding, and video understanding through the OpenAI-compatible `axllm serve` API.
- The package is intended for board-side deployment through the bundled `axllm` runtime.

## Supported Platform

- [x] AX637 / NPU1

## Validated Devices

This package has been validated on the following AX637-based device:

- AX637 / NPU1 development board

## Performance

All measurements below were taken on AX637 / NPU1. `TTFT` stands for time to first token. In this table, `TTFT` is measured end-to-end from request arrival at `axllm serve` to the first generated token, so the multimodal rows include media preprocessing and vision encoding time.

The text-only smoke prompt, `What is 1+1? Reply with the number only.`, was kept within one `128`-token prefill chunk. To avoid one-time startup effects, the text row below excludes the first request after service startup. Its `Decode` figure was measured with longer text-only generations (`max_tokens=256`) to better reflect sustained decode throughput; very short smoke replies under-report decode speed because EOS and response-tail overhead become relatively larger. The image row was measured with the packaged fixed-shape `448x448` vision encoder, `assets/sample.png`, and `What animal is the main subject in this image? Reply in Chinese only.`. The video row used the packaged sample video with `video:assets/red-panda-openai.mp4:2` and `Describe this video briefly.`.

| Scenario | Input tokens | Prefill chunks | TTFT | Decode |
|---|---:|---:|---:|---:|
| Text-only smoke prompt | `25` | `1 x 128` | `508.16 ms avg` (`507.56-508.57 ms`) | `7.24 token/s avg` |
| Image prompt | `97` | `1 x 128` | `1432.85 ms avg` (`1351.51-1593.75 ms`) | `7.33 token/s avg` |
| Video prompt | `1140` | `9 x 128` | `19469.71 ms avg` (`17852.00-22700.51 ms`) | `6.89 token/s avg` |

The packaged runtime uses the following context layout:

- `prefill_len=128`
- `kv_cache_len=1535`
- `prefill_max_token_num=1152`

`Input tokens` in the table above refers to the full request length after chat templating, not just the visual soft tokens. For the shipped `448x448` vision encoder, each selected image block contributes `64` visual soft tokens. Under the current packaged runtime settings, the sample video request in this README uses `1140` total input tokens and spans `9` prefill chunks.

### Startup Runtime Footprint

| Item | Value |
|---|---:|
| `Flash total (text + post + vision axmodels)` | `1.31 GiB` (`1344.35 MiB`) |
| `Package flash total (excluding vision_cache/)` | `1.84 GiB` (`1884.46 MiB`) |
| `Runtime CMM increment during board-side startup` | `1.40 GiB` (`1435.39 MiB`) |

The runtime CMM value above was measured during board-side startup on the validated AX637 board configuration and should be treated as a practical reference value.

## Vision Encoder Latency

Measured on AX637 / NPU1 with `/opt/bin/ax_run_model -m minicpmv4_6_vision_448.axmodel -g 0 -w 1 -r 5`.

| Model | Resolution | Soft Tokens | Time (ms) |
|---|---|---|---:|
| `minicpmv4_6_vision_448.axmodel` | `448x448` | `64` | `543.970 ms avg` |

For this packaged AX637 runtime, the visual token count is fixed by the shipped vision encoder configuration:

- `vision_width = 448`
- `vision_height = 448`
- `vision_patch_size = 14`
- patch grid = `(448 / 14) x (448 / 14) = 32 x 32`
- raw patch tokens = `32 x 32 = 1024`
- current packaged build uses the `16x` visual compression path
- `Soft Tokens = 1024 / 16 = 64`

So, for the fixed-shape runtime shipped in this repository, the relation is:

```text
Soft Tokens = (vision_width / patch_size) x (vision_height / patch_size) / 16
```

`Input tokens` in the performance table can be larger than the visual `Soft Tokens` because `axllm` counts the full templated request, including user text and chat-template tokens in addition to the visual tokens. For the packaged `assets/sample.png` request in this README, the runtime reports `input_num_token=97`, which still fits within a single `128`-token prefill chunk.

`Soft Tokens` is not a runtime-configurable value in this package. This repository ships only `minicpmv4_6_vision_448.axmodel`, so the board-side AX637 runtime always uses `448x448 -> 64` soft tokens for image encoding.

## Package Layout

```text
.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── lib/
│   └── runtime shared libraries
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   └── sample.png
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p128_l0_together.axmodel
├── ...
├── qwen3_5_text_p128_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt
```

The packaged `axllm` runtime, runtime shared libraries, and compiled `.axmodel` files live at the repository root.

## Sample Image

The `axllm` flow uses the sample image:
`assets/sample.png`

![sample](assets/sample.png)

## Sample Video

The package also includes a packaged sample video for board-side video understanding validation:

- `assets/red-panda-openai.mp4`

## Direct Inference with `axllm`

> The `axllm` workflow is still being refined. The instructions below reflect the current validated flow and may be adjusted as the packaging continues to evolve.

### Download the Model Package

Download the release package from Hugging Face:

```shell
mkdir -p AXERA-TECH/MiniCPM-V-4.6-AX637
cd AXERA-TECH/MiniCPM-V-4.6-AX637
hf download AXERA-TECH/MiniCPM-V-4.6-AX637 --local-dir .
```

### Install `axllm`

Option 1: use the validated binary included in this repository:

```bash
chmod +x ./bin/axllm
```

Option 2: install `axllm` from the public repository:

```shell
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh
```

Option 3: install with a one-line command:

```shell
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash
```

Option 4: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated `axllm` binary from GitHub Actions:
`https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm`
Then run:

```shell
chmod +x axllm
sudo mv axllm /usr/bin/axllm
```

### Run on the Board

The package root is already arranged for `axllm`, so no extra runtime path arguments are required.

For multimodal testing, you can use the packaged sample image shown above: `./assets/sample.png`, or the packaged sample video: `./assets/red-panda-openai.mp4`.

```bash
./bin/axllm run .
```

In interactive mode:

- press `Enter` directly for text-only chat
- input an image path for single-image chat
- input `video:/path/to/frames_dir` or `video:/path/to/video.mp4` for video chat

### Serve with `axllm`

From the package root on the board:

```bash
./bin/axllm serve . --port 8000
```

Expected model id:

```text
AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16
```

Health check:

```bash
curl http://127.0.0.1:8000/health
```

A typical startup log looks like this:

```text
INF Init | LLM init start
INF Init | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
INF Init | attention config: layers=24 sliding=0 full=6 linear=18 sliding_window=0 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
...
INF Init | max_token_len : 1535
INF Init | kv_cache_size : 512, kv_cache_num: 1535
INF init_groups_from_model | prefill_token_num : 128
INF init_groups_from_model | prefill_max_token_num : 1152
INF Init | MiniCPM-V-4.6 token ids: image_pad=248056 video_pad=248057
INF Init | VisionModule init ok: type=MiniCPMV46VL, tokens_per_block=64, embed_size=1024, out_dtype=fp32
INF Init | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16
```

You can then send requests to the server using the API endpoints shown in the log. For example, to check the health status and list the available models:

```bash
curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

Example output:

```json
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}
{
  "data": [
    {
      "created": 1780908633,
      "id": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}
```

![openai_api_demo](assets/openai_api_demo.png)

### Text Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "中国和美国的首都分别是哪里"}
        ]
      }
    ],
    "max_tokens": 32
  }'
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "中国的首都是北京，美国的首都是华盛顿。"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
  "object": "chat.completion"
}
```

### Image Request

```bash
python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What animal is the main subject in this image? Reply in Chinese only."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 32,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=60) as resp:
    print(resp.read().decode())
PY
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "龙虾"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
  "object": "chat.completion"
}
```

### Video Request

`axllm serve` accepts either a frames directory or a raw video file:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "video:/path/to/frames_dir"}},
          {"type": "text", "text": "Describe this video briefly."}
        ]
      }
    ],
    "max_tokens": 128
  }'
```

For a raw video file, use `video:/path/to/video.mp4`. If you need to request a specific sampling FPS, use the form `video:/path/to/video.mp4:2`.

To test the packaged sample video from the package root, you can set:

```bash
VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"
```

and then use `video:${VIDEO_PATH}:2` in the request payload.

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Two red pandas are playing with a rope. One is on a branch, the other is on the ground."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-AX637-C128-P1152-CTX1535-w8a16",
  "object": "chat.completion"
}
```

## Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

- Original Hugging Face model: [openbmb/MiniCPM-V-4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6)
- AXERA conversion and deployment workflow: [AXERA-TECH/MiniCPM-V-4.6.axera](https://github.com/AXERA-TECH/MiniCPM-V-4.6.axera)

## Discussion

- GitHub Issues
- QQ group: `139953715`