Instructions to use XiaomiMiMo/MiMo-7B-RL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XiaomiMiMo/MiMo-7B-RL with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XiaomiMiMo/MiMo-7B-RL", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("XiaomiMiMo/MiMo-7B-RL", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XiaomiMiMo/MiMo-7B-RL with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XiaomiMiMo/MiMo-7B-RL"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XiaomiMiMo/MiMo-7B-RL

SGLang

How to use XiaomiMiMo/MiMo-7B-RL with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XiaomiMiMo/MiMo-7B-RL" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XiaomiMiMo/MiMo-7B-RL" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XiaomiMiMo/MiMo-7B-RL with Docker Model Runner:
```
docker model run hf.co/XiaomiMiMo/MiMo-7B-RL
```

Is this the MIMo-7B GGUF format?

by wenguoli - opened Apr 30, 2025

Discussion

wenguoli

Apr 30, 2025

https://huggingface.co/jedisct1/MiMo-7B-RL-GGUF

When will the official release the GGUF format, or support ollama?

MrDevolver

Apr 30, 2025

https://huggingface.co/jedisct1/MiMo-7B-RL-GGUF

When will the official release the GGUF format, or support ollama?

As of now, I'd not recommend using any of the existing GGUF models created based on this model. Author of one of them explicitly stated in the model card the following:

I have deleted the mtp layers in order to make it work with llama.cpp. Quality might be degraded.

A proper implementation would be better, but this will work until that is implemented.

This means that the conversion was impossible without this "surgery", so anyone who converted this model into GGUF format ended up with either broken quants or quants which may have degraded quality.

Let's be honest, even when the model is perfectly supported by llama.cpp, the nature of the quantization prevents the model from working in its full potential. If you must remove some parts of it just to make the conversion possible, it will degrade the quality even further.

This is a small 7B model and those are the ones that usually suffer the most from quantization itself, should the quality be degraded even further, chances are it will not reach the performance even of smaller models. Until it gets proper support in llama.cpp, which usually requires full cooperation between the model author and the llama.cpp team, it is not advisable to use these models in GGUF format.

quantflex

Apr 30, 2025

Hi guys,

Yes, @MrDevolver is correct 👍 That quote is from my repo here: https://huggingface.co/quantflex/MiMo-7B-RL-nomtp-GGUF

I also put "nomtp" in the title of the repo because I wanted to make it very clear that these are not the final quants as llama.cpp doesn't have an official implementation yet.

I should say that it says this in XiaomiMiMo official repo:

Or, you can register a vLLM loader for MiMo without loading MTP parameters.

So maybe it's not too bad, but I'm not sure. It is my understanding that MTP (multi-token prediction) is used both for quality and efficiency.

quantflex

Apr 30, 2025

I made a tutorial:

https://huggingface.co/XiaomiMiMo/MiMo-7B-RL/discussions/5

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment