Instructions to use unsloth/MiniMax-M2.5-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/MiniMax-M2.5-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/MiniMax-M2.5-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/MiniMax-M2.5-GGUF", dtype="auto")

llama-cpp-python

How to use unsloth/MiniMax-M2.5-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/MiniMax-M2.5-GGUF",
	filename="BF16/MiniMax-M2.5-BF16-00001-of-00010.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use unsloth/MiniMax-M2.5-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/MiniMax-M2.5-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/MiniMax-M2.5-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MiniMax-M2.5-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/MiniMax-M2.5-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/MiniMax-M2.5-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MiniMax-M2.5-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/MiniMax-M2.5-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/MiniMax-M2.5-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/MiniMax-M2.5-GGUF with Ollama:
```
ollama run hf.co/unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
```

Unsloth Studio new

How to use unsloth/MiniMax-M2.5-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/MiniMax-M2.5-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/MiniMax-M2.5-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/MiniMax-M2.5-GGUF to start chatting

Pi new

How to use unsloth/MiniMax-M2.5-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/MiniMax-M2.5-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Docker Model Runner
How to use unsloth/MiniMax-M2.5-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/MiniMax-M2.5-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/MiniMax-M2.5-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.MiniMax-M2.5-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Q3_K_XL - Results with: 1x XTX 7900 24GB VRAM // Ryzen 7 9700X 96GB

by flaviocb - opened Feb 14

Discussion

flaviocb

Feb 14

•

edited Feb 14

Hardware:

GPU: 1x AMD Radeon RX 7900 XTX (24GB)
CPU: Ryzen 7 9700X
RAM: 96GB (2x48GB) DDR5 @ 6000 MT/s

Performance:

"timings":{"cache_n":0,"prompt_n":76,"prompt_ms":2060.365,"prompt_per_token_ms":27.11006578947368,"prompt_per_second":36.886668138897726,"predicted_n":512,"predicted_ms":42042.831,"predicted_per_token_ms":82.114904296875,"predicted_per_second":12.178057181734504}}

Small prompt (76 tokens):
Prompt Eval: ~36.9 t/s
Eval (Generation): ~12.2 t/s

"timings":
{"cache_n":0,"prompt_n":2976,"prompt_ms":25355.593,"prompt_per_token_ms":8.520024529569893,"prompt_per_second":117.37055410220538,"predicted_n":512,"predicted_ms":46464.055,"predicted_per_token_ms":90.750107421875,"predicted_per_second":11.019270702912175}}

Longer prompt (2976 tokens):
Prompt Eval: ~117.37 t/s
Eval (Generation): ~11.0 t/s

Setup:

I'm using llama.cpp (server-rocm), docker image: ghcr.io/ggml-org/llama.cpp:server-rocm
The 48k context with q8_0 KV cache takes up about 7.5GB VRAM, leaving roughly 15GB VRAM for the model layers.

Llama.cpp configuraion:

"-ot", "blk.([8-9]|[1-5][0-9]|6[0-1]).ffn_.*_exps.weight=CPU",

Memory & Context

"--no-mmap",
"--flash-attn", "on",
"-c", "49152",
"--cache-type-k", "q8_0",
"--cache-type-v", "q8_0"

My context:

The 48k context is the comfortable amount that I need for the usage I have for the model doing a number of tasks for me, that is why i choose that. This configuration also leaves some VRAM and RAM for system overhead to make sure it wont go OOM.

*Please let me know if with similar hardware setup you were able to squeeze out more performance. Comment with the config.

YellowjacketGames

Feb 16

I'm wondering if you have any insight on whether there is observable precision loss at these lower quants. I can run quant 2 with full gpu offload but i'm concerned that such low quants will end up producing nonsense. any observations you can make about that?

flaviocb

Feb 16

Unfortunately i don't. I would need to spend some time using the full model online to be able to compare.

What I can say is that the Q3_K_XL quant is working very well within the system and tools I have, and has been the best 'one-shot' coding local model I have ever run, but since my hardware is very limited I haven't ever run very big models locally.

At least there has been no strange hallucinations or bad instruction following so far. There was actually one thing, he was sometimes writing a word or two in chinese inside the text (I use it in brazilian portuguese), but that was solved with an extra line the system prompt, telling him explicitly to not use any other language other then portugues and english.

YellowjacketGames

Feb 20

Wow, if the Q3 is performing well, then I'm excited.

Question, since I actually live in Portugal, can you tell which dialect of PT it speaks? Brasiliero or European?

flaviocb

Feb 21

Probably he can speak both, i specifically ask him to answer in brazilian portuguese, also my system prompts are in english, along with the tool descriptions and so on, because the models usually work better in english, and also in average 1 english word = 1.3 tokens and portuguese (brazilian) word = 2.0 tokens; so for a 1 thousand words system prompt, that counts.

Have a try, let me know. I have been having a very good time also with qwen3-coder-next 80b-a3b

YellowjacketGames

Feb 22

I was translating my card game into Portuguese ( mostly just following the terms used in other games, not specifically EU/BR ) and i had to change the font size.

The names of the cards were twice as long! ( 1.3 vs 2.0 tokens...about right )

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment