Text Generation
GGUF
English
llama.cpp
code
coding
tool-calling
agent
mixture-of-experts
long-context
imatrix
conversational
Instructions to use jedisct1/MiMo-V2.5-coder-Q2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use jedisct1/MiMo-V2.5-coder-Q2 with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="jedisct1/MiMo-V2.5-coder-Q2", filename="MiMo-V2.5-coder-Q2-00001-of-00016.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": "What is the capital of France?" } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use jedisct1/MiMo-V2.5-coder-Q2 with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf jedisct1/MiMo-V2.5-coder-Q2 # Run inference directly in the terminal: llama-cli -hf jedisct1/MiMo-V2.5-coder-Q2
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf jedisct1/MiMo-V2.5-coder-Q2 # Run inference directly in the terminal: llama-cli -hf jedisct1/MiMo-V2.5-coder-Q2
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf jedisct1/MiMo-V2.5-coder-Q2 # Run inference directly in the terminal: ./llama-cli -hf jedisct1/MiMo-V2.5-coder-Q2
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf jedisct1/MiMo-V2.5-coder-Q2 # Run inference directly in the terminal: ./build/bin/llama-cli -hf jedisct1/MiMo-V2.5-coder-Q2
Use Docker
docker model run hf.co/jedisct1/MiMo-V2.5-coder-Q2
- LM Studio
- Jan
- vLLM
How to use jedisct1/MiMo-V2.5-coder-Q2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jedisct1/MiMo-V2.5-coder-Q2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jedisct1/MiMo-V2.5-coder-Q2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/jedisct1/MiMo-V2.5-coder-Q2
- Ollama
How to use jedisct1/MiMo-V2.5-coder-Q2 with Ollama:
ollama run hf.co/jedisct1/MiMo-V2.5-coder-Q2
- Unsloth Studio new
How to use jedisct1/MiMo-V2.5-coder-Q2 with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for jedisct1/MiMo-V2.5-coder-Q2 to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for jedisct1/MiMo-V2.5-coder-Q2 to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for jedisct1/MiMo-V2.5-coder-Q2 to start chatting
- Pi new
How to use jedisct1/MiMo-V2.5-coder-Q2 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf jedisct1/MiMo-V2.5-coder-Q2
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jedisct1/MiMo-V2.5-coder-Q2" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use jedisct1/MiMo-V2.5-coder-Q2 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama-server -hf jedisct1/MiMo-V2.5-coder-Q2
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default jedisct1/MiMo-V2.5-coder-Q2
Run Hermes
hermes
- Docker Model Runner
How to use jedisct1/MiMo-V2.5-coder-Q2 with Docker Model Runner:
docker model run hf.co/jedisct1/MiMo-V2.5-coder-Q2
- Lemonade
How to use jedisct1/MiMo-V2.5-coder-Q2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jedisct1/MiMo-V2.5-coder-Q2
Run and chat with the model
lemonade run user.MiMo-V2.5-coder-Q2-{{QUANT_TAG}}List all available models
lemonade list
Add files using upload-large-folder tool
Browse files
README.md
CHANGED
|
@@ -22,24 +22,10 @@ This is a local, self-quantized GGUF build of [XiaomiMiMo/MiMo-V2.5](https://hug
|
|
| 22 |
|
| 23 |
This quant was optimized for systems with 128 GB of memory. The default serving profile targets a 128 GB Apple Silicon machine and tries to keep the model practical at a 100,000-token context. Smaller-memory systems will likely need more aggressive CPU offload, a smaller context, or a different quant.
|
| 24 |
|
| 25 |
-
It is a text-only
|
| 26 |
-
|
| 27 |
-
## Files
|
| 28 |
-
|
| 29 |
-
The model is split into 16 GGUF shards:
|
| 30 |
-
|
| 31 |
-
```text
|
| 32 |
-
MiMo-V2.5-coder-Q2-00001-of-00016.gguf
|
| 33 |
-
...
|
| 34 |
-
MiMo-V2.5-coder-Q2-00016-of-00016.gguf
|
| 35 |
-
```
|
| 36 |
-
|
| 37 |
-
Load the first shard. llama.cpp will find the remaining shards automatically.
|
| 38 |
|
| 39 |
## Quantization
|
| 40 |
|
| 41 |
-
This artifact was quantized from the original XiaomiMiMo checkpoint, not from a third-party GGUF.
|
| 42 |
-
|
| 43 |
High-level summary:
|
| 44 |
|
| 45 |
- Quant type: `Q2_K_S`
|
|
@@ -50,17 +36,9 @@ High-level summary:
|
|
| 50 |
|
| 51 |
One tokenizer metadata fix is included so llama.cpp does not warn about the base-vocab `</s>` token at load time. MiMo's actual EOS token remains `<|im_end|>`.
|
| 52 |
|
| 53 |
-
##
|
| 54 |
-
|
| 55 |
-
The importance matrix is what makes this quant more targeted than a generic low-bit conversion.
|
| 56 |
-
|
| 57 |
-
It was built from English coding and agent-style prompts: reading files, searching code, running shell commands, editing workflows, short code review tasks, and OpenAI-compatible tool calls. The tool-call calibration included realistic argument shapes such as bounded file reads, tail reads, grep context lines, and command arrays.
|
| 58 |
-
|
| 59 |
-
That means the quantization tries to spend its limited precision budget on weights that matter for coding and structured tool use. It is not a general-purpose multilingual calibration set, and it was not designed to preserve Chinese or multimodal quality.
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
The goal is not to preserve every capability of the original model equally. This build deliberately prioritizes:
|
| 64 |
|
| 65 |
- reliable OpenAI-compatible tool calls
|
| 66 |
- coding and shell-oriented agent use
|
|
@@ -71,8 +49,6 @@ Chinese-language quality and multimodal use were not optimization targets.
|
|
| 71 |
|
| 72 |
## Serving
|
| 73 |
|
| 74 |
-
Install or build llama.cpp with `llama-server` available in `PATH`. Once the model is on Hugging Face, the usual way to run it is directly from the Hub:
|
| 75 |
-
|
| 76 |
```sh
|
| 77 |
llama-server \
|
| 78 |
-hf jedisct1/MiMo-V2.5-coder-Q2 \
|
|
@@ -193,32 +169,13 @@ llama-server \
|
|
| 193 |
|
| 194 |
For best tool-calling results:
|
| 195 |
|
| 196 |
-
- Use
|
| 197 |
-
- Keep llama.cpp built-in tools disabled unless you are specifically testing them.
|
| 198 |
- Disable model reasoning output with `--reasoning off` or `MIMO_REASONING=off`.
|
| 199 |
- Set `parallel_tool_calls` to `false` if your client supports it.
|
| 200 |
- Avoid forcing `tool_choice: required`; in testing it made malformed calls more likely.
|
| 201 |
|
| 202 |
This build was tested with Swival-style tool schemas for `read_file`, `grep`, `outline`, `run_command`, and `todo`.
|
| 203 |
|
| 204 |
-
## Local Test Results
|
| 205 |
-
|
| 206 |
-
On the local 128 GB Apple Silicon M5 setup:
|
| 207 |
-
|
| 208 |
-
- The first local `Q2_K` artifact passed only 5/7 in its best Swival-shaped serving configuration.
|
| 209 |
-
- This imatrix-backed `Q2_K_S` artifact passed 21/21 across three Swival-shaped repeat runs with fast M5 serving defaults.
|
| 210 |
-
- A Swival-style `Hello` request rendered to 4,411 prompt tokens because the client included a system prompt and tool schemas. With fast M5 serving, llama.cpp processed that payload at about 239 prompt tokens/sec. With all MoE tensors on CPU, the same class of prompt processed at about 13 prompt tokens/sec.
|
| 211 |
-
|
| 212 |
-
These are local smoke and agent-harness results, not a public benchmark suite.
|
| 213 |
-
|
| 214 |
-
## Limitations
|
| 215 |
-
|
| 216 |
-
- Text-only GGUF: no vision, video, or audio encoders.
|
| 217 |
-
- MTP blocks are omitted.
|
| 218 |
-
- The quantization is very low bit. It is intended to fit and run locally, not to match the full BF16 checkpoint.
|
| 219 |
-
- The default 100,000-token context is much smaller than MiMo-V2.5's advertised 1M training context, but much more practical on this hardware.
|
| 220 |
-
- Quality should be validated on your own coding and tool-calling workloads before relying on it.
|
| 221 |
-
|
| 222 |
## License
|
| 223 |
|
| 224 |
The upstream model card for `XiaomiMiMo/MiMo-V2.5` declares the MIT license. This derived GGUF is provided under the same license metadata.
|
|
|
|
| 22 |
|
| 23 |
This quant was optimized for systems with 128 GB of memory. The default serving profile targets a 128 GB Apple Silicon machine and tries to keep the model practical at a 100,000-token context. Smaller-memory systems will likely need more aggressive CPU offload, a smaller context, or a different quant.
|
| 24 |
|
| 25 |
+
It is a text-only quantization. The original MiMo-V2.5 checkpoint is omnimodal, but this GGUF does not include the vision or audio encoders. The MiMo multi-token prediction blocks were also omitted during conversion because normal llama.cpp generation does not currently execute them for this model.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## Quantization
|
| 28 |
|
|
|
|
|
|
|
| 29 |
High-level summary:
|
| 30 |
|
| 31 |
- Quant type: `Q2_K_S`
|
|
|
|
| 36 |
|
| 37 |
One tokenizer metadata fix is included so llama.cpp does not warn about the base-vocab `</s>` token at load time. MiMo's actual EOS token remains `<|im_end|>`.
|
| 38 |
|
| 39 |
+
## Imatrix
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
+
This build deliberately prioritizes:
|
|
|
|
|
|
|
| 42 |
|
| 43 |
- reliable OpenAI-compatible tool calls
|
| 44 |
- coding and shell-oriented agent use
|
|
|
|
| 49 |
|
| 50 |
## Serving
|
| 51 |
|
|
|
|
|
|
|
| 52 |
```sh
|
| 53 |
llama-server \
|
| 54 |
-hf jedisct1/MiMo-V2.5-coder-Q2 \
|
|
|
|
| 169 |
|
| 170 |
For best tool-calling results:
|
| 171 |
|
| 172 |
+
- Use the [Swival](https://swival.dev) harness
|
|
|
|
| 173 |
- Disable model reasoning output with `--reasoning off` or `MIMO_REASONING=off`.
|
| 174 |
- Set `parallel_tool_calls` to `false` if your client supports it.
|
| 175 |
- Avoid forcing `tool_choice: required`; in testing it made malformed calls more likely.
|
| 176 |
|
| 177 |
This build was tested with Swival-style tool schemas for `read_file`, `grep`, `outline`, `run_command`, and `todo`.
|
| 178 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
## License
|
| 180 |
|
| 181 |
The upstream model card for `XiaomiMiMo/MiMo-V2.5` declares the MIT license. This derived GGUF is provided under the same license metadata.
|