Instructions to use axteam/MiniCPM5-2B-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/MiniCPM5-2B-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/MiniCPM5-2B-FP8-dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/MiniCPM5-2B-FP8-dynamic")
model = AutoModelForCausalLM.from_pretrained("axteam/MiniCPM5-2B-FP8-dynamic", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/MiniCPM5-2B-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/MiniCPM5-2B-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/MiniCPM5-2B-FP8-dynamic

SGLang

How to use axteam/MiniCPM5-2B-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/MiniCPM5-2B-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/MiniCPM5-2B-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/MiniCPM5-2B-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/MiniCPM5-2B-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/axteam/MiniCPM5-2B-FP8-dynamic
```

MiniCPM5-2B-FP8-dynamic / README.md

uzins

Card fixes 2026-11-02: full card audit — benchmark notes and MMLU method, BF16 shares, requirements, GGUF/diffusion details

bc1afe2 verified about 5 hours ago

preview code

Raw History Blame Contribute Delete

10.7 kB

---
base_model: openbmb/MiniCPM5-2B
base_model_relation: quantized
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
  - minicpm
  - compressed-tensors
  - vllm
  - quantized
  - fp8
---
# MiniCPM5-2B · FP8 (dynamic)

8-bit quantization of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) for **vLLM serving**. Quantized by **axteam**.

| | |
|---|---|
| **Base model** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) |
| **Method** | FP8 (dynamic) |
| **Format** | compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) |
| **Weights / activations** | W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations |
| **Architecture** | Dense (Llama-architecture) · text-only |
| **Size on disk** | ~3.1 GB |
| **Calibration** | None (data-free) |
| **License** | Apache-2.0 |

## Quantization details
- **Scheme** — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
- **Kept in BF16** — token embeddings, the LM head, and all RMSNorms (about 21% of all parameters in this build).
- Post-training quantization (no retraining). Tokenizer / `chat_template` identical to the original.

## Benchmarks (measured)

Every column was measured with the same harness and settings — [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (`mmlu_generative`: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

| Benchmark | BF16 (original) | **FP8-dynamic (this repo)** | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot, generative) | 59.74 | **59.56 (99.7%)** | 59.86 (100.2%) | 56.81 (95.1%) | 53.50 (89.5%) | 56.31 (94.3%) |
| GSM8K (5-shot, strict) | 73.01 | **71.27 (97.6%)** | 73.01 (100.0%) | 65.20 (89.3%) | 55.04 (75.4%) | 65.96 (90.3%) |
| ARC-Challenge (0-shot) | 43.94 | **44.20 (100.6%)** | 43.86 (99.8%) | 43.43 (98.8%) | 40.10 (91.3%) | 44.88 (102.1%) |
| Winogrande (5-shot) | 64.40 | **63.22 (98.2%)** | 64.40 (100.0%) | 62.75 (97.4%) | 59.04 (91.7%) | 62.12 (96.4%) |
| KMMLU (5-shot, direct) | 37.65 | **38.38 (101.9%)** | 38.27 (101.7%) | 35.56 (94.4%) | 34.06 (90.5%) | 30.73 (81.6%) |
| HAE-RAE (0-shot) | 39.05 | **38.13 (97.7%)** | 39.14 (100.2%) | 35.38 (90.6%) | 30.16 (77.2%) | 36.94 (94.6%) |
| **Average** | **52.97** | **52.46 (99.0%)** | **53.09 (100.2%)** | **49.85 (94.1%)** | **45.32 (85.6%)** | **49.49 (93.4%)** |

## Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).

| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.6494 | 1.5988 (**+3.2%**) |
| Top-1 next-token agreement | 92.9% | 98.9% |

## Requirements

| | |
|---|---|
| **Runtime** | vLLM — the architecture is in vLLM; **tested here with vLLM 0.32.0**. |
| **vLLM install** | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| **GPU** | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| **Memory** | Weights: **3.1 GB**. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU. |
| **Transformers (without vLLM)** | `transformers>=4.56.0` + `compressed-tensors>=0.15.0` (checked 4.56.2 / 4.57.6 with FP8, AWQ-4bit and NVFP4A16, and 4.56.0 with AWQ-4bit; on transformers 4.x the tokenizer also returns `token_type_ids`, which the model does not accept, so drop it from the tokenizer output before calling `generate`, as these checks did). |
| **trust_remote_code** | not needed |

## Usage (vLLM)

Start an OpenAI-compatible server:

```bash
vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768
```

Query it:

```bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/MiniCPM5-2B-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "top_p": 0.95, "max_tokens": 512
}'
```

Offline inference in Python:

```python
from vllm import LLM, SamplingParams
llm = LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)
```

> Recommended sampling (from the original `generation_config.json`): `temperature=1.0`, `top_p=0.95`.
> Thinking mode is on by default in the chat template; to turn it off pass `chat_template_kwargs={"enable_thinking": False}` (Python `llm.chat(..., chat_template_kwargs=...)`, or `"chat_template_kwargs": {"enable_thinking": false}` in the request body).

## Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a **runtime** choice, independent of the
weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

```bash
vllm serve axteam/MiniCPM5-2B-FP8-dynamic --kv-cache-dtype fp8
```

## License
Quantized derivative of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

---
## 한국어 요약

**[`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B)의 FP8 (W8A8) 양자화 빌드** · 제작 axteam

### 개요
| 항목 | 내용 |
|---|---|
| 원본 모델 | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) |
| 양자화 방식 | FP8 동적 양자화 (W8A8) |
| 저장 형식 | compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) |
| 모델 구조 | Dense (Llama 구조) · 텍스트 전용 |
| 용량 | 약 3.1 GB |
| 보정 데이터 | 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산) |

### 양자화 상세
- **가중치**: FP8(E4M3), 출력 채널별 스케일
- **활성화**: 실행 시 토큰별 동적 FP8
- **양자화 절차**: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
- **BF16 유지**: 토큰 임베딩, LM 헤드, 모든 RMSNorm — 전체 파라미터의 약 21%
- **원본 파일**: 토크나이저·chat template은 원본과 동일

### 품질 (실측)
- **원본 대비 충실도**: PPL 1.6494 (원본 1.5988, +3.2%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 92.9% (원본 자체 기준값 98.9%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
- **벤치마크 평균**: 52.46 (원본 52.97, 99.0%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
- **한국어 벤치마크**: KMMLU 38.38 (원본 37.65, 101.9%) · HAE-RAE 38.13 (원본 39.05, 97.7%)
- **MMLU 방식**: lm-eval 생성형 변형(`mmlu_generative`) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가

### 요구사항
- **실행 엔진**: vLLM, 이 빌드는 vLLM 0.32.0에서 실행 확인
- **vLLM 0.32 설치 조건**: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
- **GPU**: NVIDIA Turing(sm75, 예: T4) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
- **메모리**: 가중치 3.1 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 `--tensor-parallel-size N`으로 분산
- **Transformers로 직접 사용**: transformers 4.56.0 이상 + compressed-tensors 0.15.0 이상 (4.56.2·4.57.6에서 FP8·AWQ-4bit·NVFP4A16, 4.56.0에서 AWQ-4bit 확인, transformers 4.x에서는 토크나이저가 `token_type_ids`도 돌려주는데 모델이 받지 않으므로 `generate` 호출 전에 토크나이저 출력에서 빼야 함 — 확인 때도 뺌)
- **trust_remote_code**: 불필요

### 사용법
- **서버 실행**: `vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768` — OpenAI 호환 API
- **Python**: `LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768).chat(...)` — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
- **권장 샘플링**: `temperature=1.0`, `top_p=0.95` (원본 `generation_config.json` 기준)
- **thinking 모드**: 기본 켜짐 — 끄려면 `chat_template_kwargs={"enable_thinking": False}` 전달(Python은 `llm.chat()` 인자, HTTP는 요청 본문의 `"chat_template_kwargs": {"enable_thinking": false}`)
- **KV 캐시 절감(선택)**: `vllm serve`에 `--kv-cache-dtype fp8` 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

### 라이선스
- **라이선스**: Apache-2.0 — 원본 모델의 라이선스를 그대로 따름
- **권리**: 원본 가중치·토크나이저는 OpenBMB 소유, 이 리포는 양자화 파생물

	---
	base_model: openbmb/MiniCPM5-2B
	base_model_relation: quantized
	library_name: transformers
	license: apache-2.0
	pipeline_tag: text-generation
	tags:
	- minicpm
	- compressed-tensors
	- vllm
	- quantized
	- fp8
	---
	# MiniCPM5-2B · FP8 (dynamic)

	8-bit quantization of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) for vLLM serving. Quantized by axteam.

	\| \| \|
	\|---\|---\|
	\| Base model \| [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) \|
	\| Method \| FP8 (dynamic) \|
	\| Format \| compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) \|
	\| Weights / activations \| W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations \|
	\| Architecture \| Dense (Llama-architecture) · text-only \|
	\| Size on disk \| ~3.1 GB \|
	\| Calibration \| None (data-free) \|
	\| License \| Apache-2.0 \|

	## Quantization details
	- Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
	- Kept in BF16 — token embeddings, the LM head, and all RMSNorms (about 21% of all parameters in this build).
	- Post-training quantization (no retraining). Tokenizer / `chat_template` identical to the original.

	## Benchmarks (measured)

	Every column was measured with the same harness and settings — [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (`mmlu_generative`: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

	\| Benchmark \| BF16 (original) \| FP8-dynamic (this repo) \| AWQ-8bit \| NVFP4A16 \| NVFP4 \| AWQ-4bit \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| MMLU (5-shot, generative) \| 59.74 \| 59.56 (99.7%) \| 59.86 (100.2%) \| 56.81 (95.1%) \| 53.50 (89.5%) \| 56.31 (94.3%) \|
	\| GSM8K (5-shot, strict) \| 73.01 \| 71.27 (97.6%) \| 73.01 (100.0%) \| 65.20 (89.3%) \| 55.04 (75.4%) \| 65.96 (90.3%) \|
	\| ARC-Challenge (0-shot) \| 43.94 \| 44.20 (100.6%) \| 43.86 (99.8%) \| 43.43 (98.8%) \| 40.10 (91.3%) \| 44.88 (102.1%) \|
	\| Winogrande (5-shot) \| 64.40 \| 63.22 (98.2%) \| 64.40 (100.0%) \| 62.75 (97.4%) \| 59.04 (91.7%) \| 62.12 (96.4%) \|
	\| KMMLU (5-shot, direct) \| 37.65 \| 38.38 (101.9%) \| 38.27 (101.7%) \| 35.56 (94.4%) \| 34.06 (90.5%) \| 30.73 (81.6%) \|
	\| HAE-RAE (0-shot) \| 39.05 \| 38.13 (97.7%) \| 39.14 (100.2%) \| 35.38 (90.6%) \| 30.16 (77.2%) \| 36.94 (94.6%) \|
	\| Average \| 52.97 \| 52.46 (99.0%) \| 53.09 (100.2%) \| 49.85 (94.1%) \| 45.32 (85.6%) \| 49.49 (93.4%) \|

	## Evaluation (measured)

	Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).

	\| Metric \| This model \| BF16 original \|
	\|---\|---\|---\|
	\| Perplexity (lower = closer) \| 1.6494 \| 1.5988 (+3.2%) \|
	\| Top-1 next-token agreement \| 92.9% \| 98.9% \|

	## Requirements

	\| \| \|
	\|---\|---\|
	\| Runtime \| vLLM — the architecture is in vLLM; tested here with vLLM 0.32.0. \|
	\| vLLM install \| vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). \|
	\| GPU \| NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) \|
	\| Memory \| Weights: 3.1 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU. \|
	\| Transformers (without vLLM) \| `transformers>=4.56.0` + `compressed-tensors>=0.15.0` (checked 4.56.2 / 4.57.6 with FP8, AWQ-4bit and NVFP4A16, and 4.56.0 with AWQ-4bit; on transformers 4.x the tokenizer also returns `token_type_ids`, which the model does not accept, so drop it from the tokenizer output before calling `generate`, as these checks did). \|
	\| trust_remote_code \| not needed \|

	## Usage (vLLM)

	Start an OpenAI-compatible server:

	```bash
	vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768
	```

	Query it:

	```bash
	curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
	"model": "axteam/MiniCPM5-2B-FP8-dynamic",
	"messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
	"temperature": 1.0, "top_p": 0.95, "max_tokens": 512
	}'
	```

	Offline inference in Python:

	```python
	from vllm import LLM, SamplingParams

	llm = LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768)
	params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)
	out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
	print(out[0].outputs[0].text)
	```

	> Recommended sampling (from the original `generation_config.json`): `temperature=1.0`, `top_p=0.95`.
	> Thinking mode is on by default in the chat template; to turn it off pass `chat_template_kwargs={"enable_thinking": False}` (Python `llm.chat(..., chat_template_kwargs=...)`, or `"chat_template_kwargs": {"enable_thinking": false}` in the request body).

	## Reducing KV-cache memory (optional, serve-time)

	The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the
	weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

	```bash
	vllm serve axteam/MiniCPM5-2B-FP8-dynamic --kv-cache-dtype fp8
	```

	## License
	Quantized derivative of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

	---
	## 한국어 요약

	[`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B)의 FP8 (W8A8) 양자화 빌드 · 제작 axteam

	### 개요
	\| 항목 \| 내용 \|
	\|---\|---\|
	\| 원본 모델 \| [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) \|
	\| 양자화 방식 \| FP8 동적 양자화 (W8A8) \|
	\| 저장 형식 \| compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) \|
	\| 모델 구조 \| Dense (Llama 구조) · 텍스트 전용 \|
	\| 용량 \| 약 3.1 GB \|
	\| 보정 데이터 \| 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산) \|

	### 양자화 상세
	- 가중치: FP8(E4M3), 출력 채널별 스케일
	- 활성화: 실행 시 토큰별 동적 FP8
	- 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
	- BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm — 전체 파라미터의 약 21%
	- 원본 파일: 토크나이저·chat template은 원본과 동일

	### 품질 (실측)
	- 원본 대비 충실도: PPL 1.6494 (원본 1.5988, +3.2%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 92.9% (원본 자체 기준값 98.9%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
	- 벤치마크 평균: 52.46 (원본 52.97, 99.0%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
	- 한국어 벤치마크: KMMLU 38.38 (원본 37.65, 101.9%) · HAE-RAE 38.13 (원본 39.05, 97.7%)
	- MMLU 방식: lm-eval 생성형 변형(`mmlu_generative`) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가

	### 요구사항
	- 실행 엔진: vLLM, 이 빌드는 vLLM 0.32.0에서 실행 확인
	- vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
	- GPU: NVIDIA Turing(sm75, 예: T4) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
	- 메모리: 가중치 3.1 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 `--tensor-parallel-size N`으로 분산
	- Transformers로 직접 사용: transformers 4.56.0 이상 + compressed-tensors 0.15.0 이상 (4.56.2·4.57.6에서 FP8·AWQ-4bit·NVFP4A16, 4.56.0에서 AWQ-4bit 확인, transformers 4.x에서는 토크나이저가 `token_type_ids`도 돌려주는데 모델이 받지 않으므로 `generate` 호출 전에 토크나이저 출력에서 빼야 함 — 확인 때도 뺌)
	- trust_remote_code: 불필요

	### 사용법
	- 서버 실행: `vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768` — OpenAI 호환 API
	- Python: `LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768).chat(...)` — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
	- 권장 샘플링: `temperature=1.0`, `top_p=0.95` (원본 `generation_config.json` 기준)
	- thinking 모드: 기본 켜짐 — 끄려면 `chat_template_kwargs={"enable_thinking": False}` 전달(Python은 `llm.chat()` 인자, HTTP는 요청 본문의 `"chat_template_kwargs": {"enable_thinking": false}`)
	- KV 캐시 절감(선택): `vllm serve`에 `--kv-cache-dtype fp8` 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

	### 라이선스
	- 라이선스: Apache-2.0 — 원본 모델의 라이선스를 그대로 따름
	- 권리: 원본 가중치·토크나이저는 OpenBMB 소유, 이 리포는 양자화 파생물