Instructions to use axteam/MiniCPM5-2B-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/MiniCPM5-2B-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/MiniCPM5-2B-FP8-dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/MiniCPM5-2B-FP8-dynamic")
model = AutoModelForCausalLM.from_pretrained("axteam/MiniCPM5-2B-FP8-dynamic", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/MiniCPM5-2B-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/MiniCPM5-2B-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/MiniCPM5-2B-FP8-dynamic

SGLang

How to use axteam/MiniCPM5-2B-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/MiniCPM5-2B-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/MiniCPM5-2B-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/MiniCPM5-2B-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/axteam/MiniCPM5-2B-FP8-dynamic
```

MiniCPM5-2B-FP8-dynamic / README.md

uzins

Card fixes 2026-11-01: verified corrections, requirements

c823f36 verified 25 minutes ago

preview code

Raw History Blame Contribute Delete

7.14 kB

metadata

base_model: openbmb/MiniCPM5-2B
base_model_relation: quantized
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
  - minicpm
  - compressed-tensors
  - vllm
  - quantized
  - fp8

MiniCPM5-2B · FP8 (dynamic)

8-bit quantization of openbmb/MiniCPM5-2B for vLLM serving. Quantized by axteam.


Base model	`openbmb/MiniCPM5-2B`
Method	FP8 (dynamic)
Format	compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	Dense (Llama-architecture) · text-only
Size on disk	~3.1 GB
Calibration	None (data-free)
License	Apache-2.0

Quantization details

Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
Kept in BF16 — token embeddings, the LM head, and all RMSNorms (about 21% of all parameters in this build).
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic (this repo)	AWQ-8bit	NVFP4A16	NVFP4	AWQ-4bit
MMLU (5-shot)	59.74	59.56 (99.7%)	59.86 (100.2%)	56.81 (95.1%)	53.50 (89.5%)	56.31 (94.3%)
GSM8K (5-shot, strict)	73.01	71.27 (97.6%)	73.01 (100.0%)	65.20 (89.3%)	55.04 (75.4%)	65.96 (90.3%)
ARC-Challenge (0-shot)	43.94	44.20 (100.6%)	43.86 (99.8%)	43.43 (98.8%)	40.10 (91.3%)	44.88 (102.1%)
Winogrande (5-shot)	64.40	63.22 (98.2%)	64.40 (100.0%)	62.75 (97.4%)	59.04 (91.7%)	62.12 (96.4%)
KMMLU (5-shot, direct)	37.65	38.38 (101.9%)	38.27 (101.7%)	35.56 (94.4%)	34.06 (90.5%)	30.73 (81.6%)
HAE-RAE (0-shot)	39.05	38.13 (97.7%)	39.14 (100.2%)	35.38 (90.6%)	30.16 (77.2%)	36.94 (94.6%)
Average	52.97	52.46 (99.0%)	53.09 (100.2%)	49.85 (94.1%)	45.32 (85.6%)	49.49 (93.4%)

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).

Metric	This model	BF16 original
Perplexity (lower = closer)	1.6494	1.5988 (+3.2%)
Top-1 next-token agreement	92.9%	98.9%

Requirements


Runtime	vLLM — the architecture is in vLLM; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 3.1 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=4.56.0` + `compressed-tensors>=0.15.0` (checked 4.56.0 / 4.56.2 / 4.57.6 with FP8, AWQ-4bit, NVFP4A16).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/MiniCPM5-2B-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "top_p": 0.95, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0, top_p=0.95. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/MiniCPM5-2B-FP8-dynamic --kv-cache-dtype fp8

License

Quantized derivative of openbmb/MiniCPM5-2B (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

한국어 요약

openbmb/MiniCPM5-2B 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

방식 — FP8 E4M3, 가중치 채널별 스케일(정적·보정불필요), 활성화 동적 FP8. 형식 compressed-tensors float-quantized (scheme FP8_DYNAMIC).
BF16 유지 — 토큰 임베딩·LM 헤드·모든 norm.
PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
품질(실측) — BF16 대비 perplexity 1.6494(원본 1.5988, +3.2%), top-1 92.9%(원본 98.9%). 내부 영/한 100문항.
벤치마크(실측) — MMLU · GSM8K · ARC-Challenge · Winogrande · KMMLU · HAE-RAE 평균 BF16 52.97 → 이 모델 52.46 (회복률 99.0%). lm-evaluation-harness · vLLM, 모든 형식 같은 설정.
요구사항 — vLLM (0.32.0 에서 실측); GPU Turing(sm75) 이상. FP8 연산은 Ada·Hopper·Blackwell(sm89+), 그 아래는 vLLM 이 가중치 전용 FP8 로 자동 전환; 가중치 3.1 GB + KV 캐시; Transformers 직접 사용: transformers ≥4.56.0 + compressed-tensors ≥0.15.0.
실행 — vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/MiniCPM5-2B-FP8-dynamic").chat(...) (위 예시). 권장 샘플링 temperature=1.0, top_p=0.95. thinking 끄기: chat_template_kwargs={"enable_thinking": False}.
KV 캐시 절감(런타임) — --kv-cache-dtype fp8.