README.md

---
library_name: transformers
license: apache-2.0
base_model:
  - openbmb/MiniCPM5-1B
pipeline_tag: text-generation
tags:
  - minicpm5
  - llm
  - axera
  - ax637
  - text-generation
language:
  - en
  - zh
---

# MiniCPM5-1B on AXERA AX637

Ready-to-run, text-only deployment package for
[`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) on an
AX637 aarch64 board.

- Runtime: packaged `axllm` binary with OpenAI-compatible HTTP API and CLI.
- Target: AX637.
- Context profile: `kv_cache_len=1024`, `prefill_len=128`, and maximum
  prefill capacity `896` tokens.
- Included assets: 24 decoder `.axmodel` files, post-processing `.axmodel`,
  embedding weights, tokenizer, runtime configuration, and `bin/axllm`.

## Supported Platform

- [x] AX637 aarch64 board with the AXERA runtime installed.

This is a text-only package. The packaged configuration has
`enable_thinking=false`.

## Performance

Measurements below were taken on an AX637 board with the packaged runtime.
`TTFT` means time to first generated token.

| Scenario | Input tokens | Prefill chunks | TTFT | Decode |
|---|---:|---:|---:|---:|
| Long text validation | 846 | 7 | 6600.26 ms | 6.23 tokens/s |

The long validation request generated five tokens and exercised every shipped
prefill history group: `0`, `128`, `256`, `384`, `512`, `640`, and `768`.
Actual latency depends on board memory pressure, prompt length, and output
length.

## Startup Runtime Footprint

| Item | Value |
|---|---:|
| Package size on disk | 1.7 GiB |
| Decoder layers | 24 |
| CMM used after full model startup | 966 MB |
| OS memory used by `axllm` after full model startup (RSS) | 59,856 KiB (58.5 MiB) |
| `axllm` virtual address space (`VmSize`, mostly mmap mappings) | 1,979,888 KiB (1.89 GiB) |

The CMM figure is the AX Engine CMM-pool delta measured from before startup to
after all 24 decoder layers and the post model were loaded. The OS-memory
figure is the board-side process resident set size (`VmRSS`) after the same
startup point. `VmSize` is shown separately because the package uses memory
mapping; it is virtual address space, not physical OS memory. CMM and RSS are
the startup consumption values to use when budgeting a board, while actual
system availability depends on other workloads.

## Context Layout

| Setting | Packaged value |
|---|---:|
| KV cache length | 1024 tokens |
| Prefill chunk length | 128 tokens |
| Maximum prefill length | 896 tokens |
| Prefill history capacities | 0, 128, 256, 384, 512, 640, 768 |

Prompts longer than 128 tokens are split into chunks. The runner selects the
smallest compatible prefill group for each chunk. Leave room inside the
1024-token KV window for generated tokens when sending long prompts.

## Package Layout

```text
.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── config.json
├── post_config.json
├── minicpm5_tokenizer.txt
├── model.embed_tokens.weight.bfloat16.bin
├── llama_p128_l0_together.axmodel
├── ...
├── llama_p128_l23_together.axmodel
└── llama_post.axmodel
```

This is a flat runtime package. Run `axllm` from the package root; it reads the
root-level tokenizer, configuration, embedding, and `.axmodel` files directly.

## Download the Package

Download this repository on the host that will transfer or mount it on the
board:

```shell
mkdir -p AXERA-TECH/MiniCPM5-1B-AX637
cd AXERA-TECH/MiniCPM5-1B-AX637
hf download AXERA-TECH/MiniCPM5-1B-AX637 --local-dir .
```

## Run on the Board

The package includes a validated AX637 `axllm` binary. From the package root:

```bash
chmod +x ./bin/axllm
export LD_LIBRARY_PATH=/opt/lib:${LD_LIBRARY_PATH:-}
./bin/axllm serve . --port 8000
```

The service exposes:

```text
GET  http://<board-ip>:8000/health
GET  http://<board-ip>:8000/v1/models
POST http://<board-ip>:8000/v1/chat/completions
```

Expected model identifier:

```text
AXERA-TECH/MiniCPM5-1B-AX637-C128-P896-CTX1024
```

Verify readiness:

```bash
curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

Example health response:

```json
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}
```

### Text Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM5-1B-AX637-C128-P896-CTX1024",
    "messages": [
      {
        "role": "user",
        "content": "中国的首都是哪里？请只回答城市名。"
      }
    ],
    "max_tokens": 32,
    "temperature": 0
  }'
```

The response uses the standard OpenAI chat-completions JSON shape. Set the
OpenAI client base URL to `http://<board-ip>:8000/v1` and use the model
identifier shown above.

### Interactive CLI

```bash
export LD_LIBRARY_PATH=/opt/lib:${LD_LIBRARY_PATH:-}
./bin/axllm run .
```

Type `/q` or `/exit` to leave the interactive session.

## Conversion References

If you need the original model files or want to rebuild the deployment
artifacts, start with:

- Original Hugging Face model:
  [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B)
- AXERA conversion and deployment workflow:
  [AXERA-TECH/MiniCPM5-1B.axera](https://github.com/AXERA-TECH/MiniCPM5-1B.axera)

## Discussion

- [GitHub Issues](https://github.com/AXERA-TECH/ax-llm/issues)
- QQ group: `139953715`