Instructions to use openbmb/MiniCPM-V-2_6-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/MiniCPM-V-2_6-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="openbmb/MiniCPM-V-2_6-gguf",
	filename="ggml-model-IQ3_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use openbmb/MiniCPM-V-2_6-gguf with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

Use Docker

docker model run hf.co/openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

LM Studio
Jan
Ollama
How to use openbmb/MiniCPM-V-2_6-gguf with Ollama:
```
ollama run hf.co/openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
```

Unsloth Studio new

How to use openbmb/MiniCPM-V-2_6-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for openbmb/MiniCPM-V-2_6-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for openbmb/MiniCPM-V-2_6-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for openbmb/MiniCPM-V-2_6-gguf to start chatting

Docker Model Runner
How to use openbmb/MiniCPM-V-2_6-gguf with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM-V-2_6-gguf:Q4_K_M
```

Lemonade

How to use openbmb/MiniCPM-V-2_6-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull openbmb/MiniCPM-V-2_6-gguf:Q4_K_M

Run and chat with the model

lemonade run user.MiniCPM-V-2_6-gguf-Q4_K_M

List all available models

lemonade list

tc-mb commited on Aug 8, 2024

Commit

930fe99

verified ·

1 Parent(s): 7925292

Upload 21 files

Browse files

Files changed (22) hide show

.gitattributes +21 -0
ggml-model-IQ3_M.gguf +3 -0
ggml-model-IQ3_S.gguf +3 -0
ggml-model-IQ3_XS.gguf +3 -0
ggml-model-IQ4_NL.gguf +3 -0
ggml-model-IQ4_XS.gguf +3 -0
ggml-model-Q2_K.gguf +3 -0
ggml-model-Q3_K.gguf +3 -0
ggml-model-Q3_K_L.gguf +3 -0
ggml-model-Q3_K_M.gguf +3 -0
ggml-model-Q3_K_S.gguf +3 -0
ggml-model-Q4_0.gguf +3 -0
ggml-model-Q4_1.gguf +3 -0
ggml-model-Q4_K.gguf +3 -0
ggml-model-Q4_K_S.gguf +3 -0
ggml-model-Q5_0.gguf +3 -0
ggml-model-Q5_1.gguf +3 -0
ggml-model-Q5_K.gguf +3 -0
ggml-model-Q5_K_M.gguf +3 -0
ggml-model-Q5_K_S.gguf +3 -0
ggml-model-Q6_K.gguf +3 -0
ggml-model-Q8_0.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -36,3 +36,24 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 ggml-model-f16.gguf filter=lfs diff=lfs merge=lfs -text
 ggml-model-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 mmproj-model-f16.gguf filter=lfs diff=lfs merge=lfs -text

 ggml-model-f16.gguf filter=lfs diff=lfs merge=lfs -text
 ggml-model-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 mmproj-model-f16.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q4_K.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+ggml-model-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

ggml-model-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1cf78997419de1094e3ffa1f71aef9f13abfb531c6ace2d29bdd4d226cabcb66
+size 3572217184

ggml-model-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2a8a21308c7a6b4deda7cfd94e2128207b37797a0a2727ce0b5bfc2990b07d62
+size 3497397600

ggml-model-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7e92e3e1b88079ba5f262a92dc6e638729f5039666c4606af99b1f6638152923
+size 3344461152

ggml-model-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:37367ae200f107c1095145de243ee38e2c3efa8ae93c5e376098d50c0ae69984
+size 4461289792

ggml-model-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ba1e23aa5470520c8c920d7944479133af9ae091a23653529be4348f77ba1e18
+size 4248358752

ggml-model-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cec163c9ae08fbac5f65dcf05b80fd717acc4ac7bc631b28c0adb3ec4948e7fd
+size 3014290240

ggml-model-Q3_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5a050acc927498ca4e05d38207bbd89d013154a7b9b24ac9ff4e45c870bc7f62
+size 3806596448

ggml-model-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:970be5beebeb77298e7390b840f9be7e4749a47190b8fd1b7ddb7e30a8d43099
+size 4086664544

ggml-model-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5a050acc927498ca4e05d38207bbd89d013154a7b9b24ac9ff4e45c870bc7f62
+size 3806596448

ggml-model-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ad5363fdd5bff45ad840a1b179e8f32294d9a9e5a53902f5eca4458c868055a2
+size 3490573664

ggml-model-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:262843d4806aeb402336980badd414a72576b20b1e5d537647da15f16c4a4df0
+size 4429406528

ggml-model-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:12e6b9a27b91297f8ee275b2ac0ca796e527cbbaee1c1de22fcab72541581495
+size 4871210240

ggml-model-Q4_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a4078d53b46f22989adbf998ce5a3fd090b6541f112d7e936eb4204a04100b1
+size 4681089344

ggml-model-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6f8e13697c94a6ffdd485d4054b493a43640d5220303ee9600dd8dab4c3da564
+size 4455784768

ggml-model-Q5_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6c58f333a60ccd839d7e78d6baa7c800119f6a4c3fad220daecec5aae3a45960
+size 5313013952

ggml-model-Q5_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:32b4c781907a067b5c17e2fe2eb46597f8a5b11cb200d570126bb4651fdd8dd6
+size 5754817664

ggml-model-Q5_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3d7bad99e6548783dbc35d87cc0e04095cfa63eec985ddbbaee3122f94de6913
+size 5442668736

ggml-model-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3d7bad99e6548783dbc35d87cc0e04095cfa63eec985ddbbaee3122f94de6913
+size 5442668736

ggml-model-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:278092401a8159aa61f2fce7f39af09e51d0b5ee3a12ccfc5b1f2088b3671392
+size 5313013952

ggml-model-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5f93a5c114e396d65e76f3350508c291af8c9a89508ddf88d32a2263cf21a590
+size 6251846848

ggml-model-Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1dcd11b83b5e4922b51ac13bb21f40e808d702be056d949d681302c9999c0730
+size 8095482304