Instructions to use openbmb/MiniCPM5-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM5-2B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openbmb/MiniCPM5-2B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-2B")
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM5-2B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openbmb/MiniCPM5-2B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM5-2B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM5-2B

SGLang

How to use openbmb/MiniCPM5-2B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM5-2B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM5-2B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM5-2B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM5-2B with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM5-2B
```

FastFlowLM / Q4NX build for AMD XDNA 2 NPU (63.6 tok/s @ 2–4W on Strix Halo & Point)

by julianmb - opened about 2 hours ago

Discussion

julianmb

about 2 hours ago

Hugging Face CLI

Hi OpenBMB team & community,

We wanted to share an unofficial native port of openbmb/MiniCPM5-2B running entirely on the AMD XDNA 2 NPU (/dev/accel/accel0) via AMD FastFlowLM.

Weights & Precompiled AIE Kernels: huggingface.co/julianmb/MiniCPM5-2B-NPU2
Porting Scripts & Quality Eval: github.com/julianmb/npuhalo/tree/main/ports/minicpm5-2b

Performance on AMD Hardware (Ryzen AI Max+ 395 / Strix Halo & Strix Point)

Sustained Decode: 63.1 – 63.6 tok/s on XDNA 2 (no GPU/CPU decode required)
Prefill Speed (TTFT): 81.5 – 128.1 tok/s (~420 ms TTFT)
Active Power: ~2–4 W (leaves the 45–65W iGPU and 16 CPU cores completely idle/free)
Footprint: ~1.88 GB in NPU memory (Q4_1 / Q4NX format)
Accuracy & Reasoning: Tested across multi-step math reasoning, Python bug fixing, and structured JSON output with zero degradation compared to baseline.

How We Solved the GQA Firmware Incompatibility

MiniCPM5-2B uses 16 Query heads and 2 Key/Value heads (16:2 = 8:1 GQA ratio). AMD FastFlowLM's AIE firmware currently lacks a native 8:1 kernel for d_head=128:

4x KV Head Replication: We replicated the 2 KV heads 4x along dimension 0 into 8 KV heads (16:8 = 2:1 GQA ratio). Under Grouped Query Attention, this maintains exact bit-for-bit mathematical equivalence while matching the native _gen_mha_seq_d128_q2 AIE kernel.
Qwen3 Runtime Engine Routing: Routed execution through FastFlowLM's Qwen3 engine (libqwen3_npu.so) to dynamically dispatch d_head=128 when intermediate_size == 6144.
Identity QK-Norm Injection: Injected synthetic unit RMSNorm tensors across all 42 layers in model.q4nx, making RMSNorm a transparent identity op.

How to Run Locally

# 1. Clone the NPU weights
mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

# 2. Register under ~/.config/flm/model_list.json
# Add under "models":
# "minicpm5:2b": {
#   "path": "~/.config/flm/models/MiniCPM5-2B-NPU2",
#   "model_type": "qwen3",
#   "tokenizer": "~/.config/flm/models/MiniCPM5-2B-NPU2"
# }

# 3. Serve via FastFlowLM
flm serve minicpm5:2b --host 127.0.0.1 --port 8001

Thank you for releasing MiniCPM5-2B — its quality-to-size ratio makes it by far the strongest 2B-class model running on AMD NPUs today!

OpenBMB-LLM

OpenBMB org 40 minutes ago

Thanks for sharing this — the write-up on the GQA workaround is the part we appreciate most. Replicating the 2 KV heads 4x to hit the native 2:1 kernel is a neat way around the missing 8:1 path, and it's good to see you called out that it's mathematically equivalent rather than an approximation. The identity QK-Norm injection across all 42 layers is a clever trick too.

63 tok/s at 2–4 W is a striking number for a 2B model, especially with the iGPU and CPU left idle. Thanks also for publishing the porting scripts and eval alongside the weights — that makes the work reproducible for anyone else targeting XDNA 2.

Nice work, and glad the model is a good fit for that hardware.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment