Instructions to use openbmb/MiniCPM-Llama3-V-2_5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM-Llama3-V-2_5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openbmb/MiniCPM-Llama3-V-2_5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM-Llama3-V-2_5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM-Llama3-V-2_5

SGLang

How to use openbmb/MiniCPM-Llama3-V-2_5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM-Llama3-V-2_5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM-Llama3-V-2_5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM-Llama3-V-2_5 with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM-Llama3-V-2_5
```

咱的模型在oobabooga 的text-generation-webui上推理出错的问题

#31

by Wallis2000 - opened Jun 10, 2024

Discussion

Wallis2000

Jun 10, 2024

•

edited Jun 10, 2024

背景信息
模型所有文件都已经被下载且验证，开启了trust-remote-code，载入器：Transformers，load in 4 bit，use_double_quant，compute_dtype：float16，quant_type ：nf4

问题
在执行推理时，推理失败且报错如下：
Traceback (most recent call last):
File "/content/text-generation-webui/modules/callbacks.py", line 61, in gentask
ret = self.mfunc(callback=_callback, *args, **self.kwargs)
File "/content/text-generation-webui/modules/text_generation.py", line 376, in generate_with_callback
shared.model.generate(**kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/openbmb_MiniCPM-Llama3-V-2_5/modeling_minicpmv.py", line 327, in generate
assert input_id_list is not None
AssertionError
Output generated in 0.81 seconds (0.00 tokens/s, 0 tokens, context 59, seed 1890816148)

尝试解决发现，这个input_id_list需要模型的tokenizer将输入文本token化并传值给它，可能涉及到改动webui的代码，然后就不会了……

我只是一个普通本科软工专业学生，并不是AI专业领域的开发者。但我很喜欢研究一些好玩的ai产品，还请大佬们能多多指教下😭

Wallis2000 changed discussion title from 请问咱的模型不能在oobabooga 的text-generation-webui上运行是为什么呢？ to 咱的模型不能在oobabooga 的text-generation-webui上运行 Jun 10, 2024

Wallis2000 changed discussion title from 咱的模型不能在oobabooga 的text-generation-webui上运行 to 咱的模型在oobabooga 的text-generation-webui上推理出错的问题 Jun 10, 2024

lianxu

Jun 11, 2024

没遇到过，试试gguf版本的呢？
https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf
Q4 KM大小在我32GB RAM + RTX 3070上运行的没啥问题

Wallis2000

Jun 13, 2024

•

edited Jun 13, 2024

没遇到过，试试gguf版本的呢？
https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf
Q4 KM大小在我32GB RAM + RTX 3070上运行的没啥问题

好吧，我是是在Colab上运行的，等会儿我实验下看看

Wallis2000 changed discussion status to closed Jun 13, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment