Instructions to use openbmb/MiniCPM-2B-sft-fp32 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM-2B-sft-fp32 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openbmb/MiniCPM-2B-sft-fp32", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("openbmb/MiniCPM-2B-sft-fp32", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use openbmb/MiniCPM-2B-sft-fp32 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM-2B-sft-fp32"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-2B-sft-fp32",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM-2B-sft-fp32

SGLang

How to use openbmb/MiniCPM-2B-sft-fp32 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM-2B-sft-fp32" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-2B-sft-fp32",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM-2B-sft-fp32" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-2B-sft-fp32",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM-2B-sft-fp32 with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM-2B-sft-fp32
```

Why a 2B model size up to 80% of 7B?

by lucasjin - opened Feb 4, 2024

Discussion

lucasjin

Feb 4, 2024

Your model are very big compare with llama7b if it has 2b size, much more big than phi2

lucasjin changed discussion title from Why a 2B model size update 80% of 7B? to Why a 2B model size up to 80% of 7B? Feb 4, 2024

YaTharThShaRma999

Feb 12, 2024

@lucasjin its probably because this is at fp32 precision while llama 7b models are usually at fp16 precision(originally they probably are fp32)
fp32 is slightly better for training and its very slightly higher quality(not measurable really) but fp16 is 2x faster and 2x smaller.
you can very easily convert this to fp16 by doing
model.half()

or you can just simply download the some fp16 version instead of the fp32 version.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment