Instructions to use openbmb/MiniCPM-Llama3-V-2_5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM-Llama3-V-2_5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("openbmb/MiniCPM-Llama3-V-2_5", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use openbmb/MiniCPM-Llama3-V-2_5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/MiniCPM-Llama3-V-2_5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/MiniCPM-Llama3-V-2_5

SGLang

How to use openbmb/MiniCPM-Llama3-V-2_5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/MiniCPM-Llama3-V-2_5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/MiniCPM-Llama3-V-2_5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/MiniCPM-Llama3-V-2_5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use openbmb/MiniCPM-Llama3-V-2_5 with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM-Llama3-V-2_5
```

use with llama-cpp-python server

#70

by SurtMcGert - opened Aug 22, 2024

Discussion

SurtMcGert

Aug 22, 2024

I have been looking for an open source model capable of understanding and extracting information from images of documents that include ticks and crosses and hand written notes etc... This model is the only one capable of anything decent. As such, I wanted to set the model up as a server. Through attempting this I have discovered the world of llama-cpp and gguf files etc and I can see that this model is supported by llama-cpp. I wanted to try getting the model loaded using llama-cpp-python[server] in a docker image which I have no successfully done, however I am not getting particularly good results and I am certain its because I have set something up wrong somewhere. Since I am new to all this I cant really work out what I am doing wrong.

this is my config file for the model:
{
"host": "0.0.0.0",
"port": 2080,
"models": [
{
"model": "./model/ggml-model-Q6_K.gguf",
"model_alias": "MiniCPM",
"chat_format": "llava-1-5",
"clip_model_path": "./model/mmproj-model-f16.gguf",
"n_gpu_layers": -1,
"n_threads": -1,
"n_batch": 512,
"offload_kqv": true,
"n_ctx": 2048
}
]
}

this is the command I use to launch the server
python3 -m llama_cpp.server --config_file $CONFIG_FILE

and then I make a request using the openAI api like so:
def image_to_base64_data_uri(file_path):
with open(file_path, "rb") as img_file:
base64_data = base64.b64encode(img_file.read()).decode('utf-8')
return f"data:image/png;base64,{base64_data}"

prompt = "Using only a json format, give me the list of vehicle checks in this document that are not applicable."
file_path = 'test_images/service.jpg'
data_uri = image_to_base64_data_uri(file_path)

response = client.chat.completions.create(
model="MiniCPM",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": data_uri
},
},
{"type": "text", "text": prompt},
],
}
],
)
print(response.choices[0].message)

however, I am getting poor responses from the server, much poorer than even when I just tried the Q4 model in kaggle using the steps outlined here
https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv

some examples of the responses I have got are as follows: (these were all from using the prompt I have in the above example code)
ChatCompletionMessage(content='3.0.7.0.70.5.2.4.9.8.3.1.6.0.', refusal=None, role='assistant', function_call=None, tool_calls=None)

ChatCompletionMessage(content=' The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check level", "Power windows - Up/Down", "Mirrors - Adjusting side mirrors", "Wipers - Test front wiper blade", "Lights - Rear lights", and "Tires - Check tyre pressure".ASSISTANT: The following vehicle checks in the document are not applicable: "Check steering", "Automatic Transmission - Check level", "Coolant system - Check', refusal=None, role='assistant', function_call=None, tool_calls=None)

so sometimes just a list of numbers, sometimes repeating itself a lot with content that isnt even present in the image. I know the model can do what I am looking for because the live demo that you can use to test the full sized model worked excellently, and even the Q4 model was pretty good but ever so slightly off which is why im now trying the Q6 model. But these responses are completely off so I dont know what I have set up wrong.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment