Instructions to use openbmb/MiniCPM-o-2_6-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use openbmb/MiniCPM-o-2_6-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Use Docker

docker model run hf.co/openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

LM Studio
Jan
Ollama
How to use openbmb/MiniCPM-o-2_6-gguf with Ollama:
```
ollama run hf.co/openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
```

Unsloth Studio

How to use openbmb/MiniCPM-o-2_6-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for openbmb/MiniCPM-o-2_6-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for openbmb/MiniCPM-o-2_6-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for openbmb/MiniCPM-o-2_6-gguf to start chatting

How to use openbmb/MiniCPM-o-2_6-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "openbmb/MiniCPM-o-2_6-gguf:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use openbmb/MiniCPM-o-2_6-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use openbmb/MiniCPM-o-2_6-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "openbmb/MiniCPM-o-2_6-gguf:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use openbmb/MiniCPM-o-2_6-gguf with Docker Model Runner:
```
docker model run hf.co/openbmb/MiniCPM-o-2_6-gguf:Q4_K_M
```

Lemonade

How to use openbmb/MiniCPM-o-2_6-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull openbmb/MiniCPM-o-2_6-gguf:Q4_K_M

Run and chat with the model

lemonade run user.MiniCPM-o-2_6-gguf-Q4_K_M

List all available models

lemonade list

why MiniCPM-o-2.6 show info that is qwen2

by orca-zhang - opened Feb 12, 2025

Discussion

orca-zhang

Feb 12, 2025

[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 0: general.architecture str = qwen2
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 1: general.type str = model
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 2: general.name str = Model
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 3: general.size_label str = 7.6B
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 4: qwen2.block_count u32 = 28
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 5: qwen2.context_length u32 = 32768
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 6: qwen2.embedding_length u32 = 3584
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 7: qwen2.feed_forward_length u32 = 18944
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 8: qwen2.attention.head_count u32 = 28
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 9: qwen2.attention.head_count_kv u32 = 4
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 10: qwen2.rope.freq_base f32 = 1000000.000000
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 11: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 12: tokenizer.ggml.model str = gpt2
[2025-02-12 11:48:10.368] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 13: tokenizer.ggml.pre str = qwen2
[2025-02-12 11:48:10.387] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,151700] = ["!", """, "#", "$", "%", "&", "'", ...
[2025-02-12 11:48:10.396] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,151700] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 151644
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 151645
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 19: tokenizer.ggml.unknown_token_id u32 = 128244
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 151643
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = false
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 22: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 23: general.quantization_version u32 = 2
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - kv 24: general.file_type u32 = 2
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - type f32: 141 tensors
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q4_0: 197 tensors
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: llama_model_loader: - type q6_K: 1 tensors
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: print_info: file format = GGUF V3 (latest)
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: print_info: file type = Q4_0
[2025-02-12 11:48:10.414] [info] [WASI-NN] llama.cpp: print_info: file size = 4.12 GiB (4.65 BPW)
[2025-02-12 11:48:10.525] [info] [WASI-NN] llama.cpp: load: special tokens cache size = 58
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: load: token to piece cache size = 0.9313 MB
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: arch = qwen2
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: vocab_only = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_ctx_train = 32768
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_embd = 3584
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_layer = 28
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_head = 28
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_head_kv = 4
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_rot = 128
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_swa = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_k = 128
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_embd_head_v = 128
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_gqa = 7
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_embd_k_gqa = 512
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_embd_v_gqa = 512
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: f_norm_eps = 0.0e+00
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: f_norm_rms_eps = 1.0e-06
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: f_clamp_kqv = 0.0e+00
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: f_max_alibi_bias = 0.0e+00
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: f_logit_scale = 0.0e+00
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_ff = 18944
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_expert = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_expert_used = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: causal attn = 1
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: pooling type = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: rope type = 2
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: rope scaling = linear
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: freq_base_train = 1000000.0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: freq_scale_train = 1
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_ctx_orig_yarn = 32768
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: rope_finetuned = unknown
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: ssm_d_conv = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: ssm_d_inner = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: ssm_d_state = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_rank = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: ssm_dt_b_c_rms = 0
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: model type = 7B
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: model params = 7.61 B
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: general.name = Model
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: vocab type = BPE
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_vocab = 151700
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: n_merges = 151387
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: BOS token = 151644 '<|im_start|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOS token = 151645 '<|im_end|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOT token = 151645 '<|im_end|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: UNK token = 128244 ''
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: PAD token = 151643 '<|endoftext|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: LF token = 198 'Ċ'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM PRE token = 151659 '<|fim_prefix|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM SUF token = 151661 '<|fim_suffix|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM MID token = 151660 '<|fim_middle|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM PAD token = 151662 '<|fim_pad|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM REP token = 151663 '<|repo_name|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: FIM SEP token = 151664 '<|file_sep|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOG token = 151643 '<|endoftext|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOG token = 151645 '<|im_end|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOG token = 151662 '<|fim_pad|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOG token = 151663 '<|repo_name|>'
[2025-02-12 11:48:10.571] [info] [WASI-NN] llama.cpp: print_info: EOG token = 151664 '<|file_sep|>'

tokenizer.chat_template also match qwen2's not like minicpm-v

tc-mb

OpenBMB org Feb 12, 2025

The llm part of the model is qwen2.5.

aniketneverbrokeagain

Feb 12, 2025

can we use this latest model on ollama , i am trying for the past few days but no sucess at all

orca-zhang

Feb 19, 2025

can we use this latest model on ollama , i am trying for the past few days but no sucess at all

I can use with the llama.cpp series toolkit normally. If the download fails, you can try multiple times. If it is a running error, it is recommended to consider changing the ollama version.

orca-zhang changed discussion status to closed Feb 19, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment