Instructions to use XiaomiMiMo/MiMo-7B-RL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XiaomiMiMo/MiMo-7B-RL with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XiaomiMiMo/MiMo-7B-RL", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("XiaomiMiMo/MiMo-7B-RL", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XiaomiMiMo/MiMo-7B-RL with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XiaomiMiMo/MiMo-7B-RL"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XiaomiMiMo/MiMo-7B-RL

SGLang

How to use XiaomiMiMo/MiMo-7B-RL with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XiaomiMiMo/MiMo-7B-RL" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XiaomiMiMo/MiMo-7B-RL" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XiaomiMiMo/MiMo-7B-RL",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XiaomiMiMo/MiMo-7B-RL with Docker Model Runner:
```
docker model run hf.co/XiaomiMiMo/MiMo-7B-RL
```

Tutorial on making GGUF quants for this model (until proper support is implemented)

by quantflex - opened Apr 30, 2025

Discussion

quantflex

Apr 30, 2025

Here is how you can make your own gguf's if anyone is interested (without mtp layers)
You can also find them premade in my repo here: https://huggingface.co/quantflex/MiMo-7B-RL-nomtp-GGUF

NOTE: This should be a temporary solution until we have proper llama.cpp support. As it might degrade quality.

First in config.json change this (on line 3):

    "MiMoForCausalLM"

to:

    "Qwen2ForCausalLM"

This is done because it's practically based on Qwen2 architecture with very little modifications.

After that save this as a python script in the safetensors directory: deletemtp.py

from safetensors import safe_open
from safetensors.torch import save_file

shard_path = "model-00004-of-00004.safetensors"

with safe_open(shard_path, framework="pt") as f:
    tensors = {
        k: f.get_tensor(k)
        for k in f.keys()
        if not k.startswith("model.mtp_layers.0.")
    }

save_file(tensors, shard_path)

This will delete the mtp layers. So just run the script with python.

And finally, in model.safetensors.index.json delete all the lines that have the word mtp in them:

    "model.mtp_layers.0.final_layernorm.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.hidden_layernorm.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.input_layernorm.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.input_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
    "model.mtp_layers.0.token_layernorm.weight": "model-00004-of-00004.safetensors",

Then convert the model to gguf as usual with convert_hf_to_gguf.py

ngxson

May 1, 2025

@quantflex hi, could you please temporary remove this guide please?

This will produce wrong GGUF which does not support MTP, the support will be added later: https://github.com/ggml-org/llama.cpp/pull/13236

Once it is added, model produced using your guide will not be compatible

ngxson

May 1, 2025

•

edited May 1, 2025

Edit: hmm ok maybe you can keep this for a while, as this is the same situation with deepseek v3 (I completely forgot that it also has MTP, but the GGUF don't support MTP)

But it's up to you anw. Personally I think the 7B is small enough so maybe more people will help finishing the PR

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment