Instructions to use Inferact/MiniMax-M3-EAGLE3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Inferact/MiniMax-M3-EAGLE3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Inferact/MiniMax-M3-EAGLE3")

# Load model directly
from transformers import AutoTokenizer, LlamaForCausalLMEagle3

tokenizer = AutoTokenizer.from_pretrained("Inferact/MiniMax-M3-EAGLE3")
model = LlamaForCausalLMEagle3.from_pretrained("Inferact/MiniMax-M3-EAGLE3")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Inferact/MiniMax-M3-EAGLE3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Inferact/MiniMax-M3-EAGLE3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/MiniMax-M3-EAGLE3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Inferact/MiniMax-M3-EAGLE3

SGLang

How to use Inferact/MiniMax-M3-EAGLE3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Inferact/MiniMax-M3-EAGLE3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/MiniMax-M3-EAGLE3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Inferact/MiniMax-M3-EAGLE3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/MiniMax-M3-EAGLE3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Inferact/MiniMax-M3-EAGLE3 with Docker Model Runner:
```
docker model run hf.co/Inferact/MiniMax-M3-EAGLE3
```

rogerwyf commited on 25 days ago

Commit

454f633

verified ·

1 Parent(s): 56260f0

Update README.md

Browse files

Files changed (1) hide show

README.md +4 -26

README.md CHANGED Viewed

@@ -14,7 +14,7 @@ tags:
 ## Model Overview
-**Inferact/MiniMax-M3-EAGLE3** is an EAGLE3 draft model for accelerating inference of [MiniMax-M3](https://huggingface.co/MiniMaxAI/Minimax-M3-preview). It is served end-to-end with **[vLLM](https://github.com/vllm-project/vllm)** and was trained using **[TorchSpec](https://github.com/lightseekorg/TorchSpec)** — a torch-native online speculative-decoding training framework that runs FSDP training and vLLM-based target inference concurrently, learning from **MiniMax-M3-regenerated responses and live vLLM-generated hidden states** to match the base model's exact token distribution.
 The draft is a **1-layer** dense Llama (`LlamaForCausalLMEagle3`, ~3.3 B params) operating on MiniMax-M3's `hidden_size=6144` / `vocab_size=200064`; at serve time it shares the target's embedding and LM head (EAGLE3). See `config.json` for the full architecture.
@@ -22,7 +22,7 @@ The draft is a **1-layer** dense Llama (`LlamaForCausalLMEagle3`, ~3.3 B params)
 ## Performance
-All numbers are measured end-to-end against `Inferact/minimax-m3-final` (MXFP8) served with vLLM at `tensor-parallel-size=4`, `num_speculative_tokens=3`, and `--enforce-eager`. Greedy draft sampling (`topk=1`).
 | Category | Dataset | n | Mean Accept Length | Draft Accept Rate | Per-pos Accept Rate |
 |---|---|---:|---:|---:|---|
@@ -68,31 +68,9 @@ Draft architecture, TTT depth, sequence length, cluster layout, and optimizer ar
 ### Launch Server (vLLM)
 ```bash
-vllm serve Inferact/minimax-m3-final \
   --tensor-parallel-size 4 \
   --gpu-memory-utilization 0.90 \
-  --max-model-len 65536 \
   --block-size 128 \
-  --enforce-eager \
-  --no-enable-prefix-caching \
   --speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
-```
-### Run Benchmarks
-```bash
-vllm-bench \
-  --backend openai-chat \
-  --base-url http://127.0.0.1:8000 \
-  --model Inferact/minimax-m3-final \
-  --dataset-name speed-bench \
-  --speed-bench-config throughput_16k \
-  --speed-bench-max-input-len 10240 \
-  --speed-bench-category low_entropy \
-  --num-warmups 5 \
-  --num-prompts 1000 \
-  --output-len 1536 \
-  --sweep-max-concurrency 64 \
-  --sweep-num-prompts-factor 1 \
-  --save-result
-```

 ## Model Overview
+**Inferact/MiniMax-M3-EAGLE3** is an EAGLE3 draft model for accelerating inference of [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3). It is served end-to-end with **[vLLM](https://github.com/vllm-project/vllm)** and was trained using **[TorchSpec](https://github.com/lightseekorg/TorchSpec)** — a torch-native online speculative-decoding training framework that runs FSDP training and vLLM-based target inference concurrently, learning from **MiniMax-M3-regenerated responses and live vLLM-generated hidden states** to match the base model's exact token distribution.
 The draft is a **1-layer** dense Llama (`LlamaForCausalLMEagle3`, ~3.3 B params) operating on MiniMax-M3's `hidden_size=6144` / `vocab_size=200064`; at serve time it shares the target's embedding and LM head (EAGLE3). See `config.json` for the full architecture.
 ## Performance
+All numbers are measured end-to-end against `MiniMaxAI/MiniMax-M3-MXFP8` served with vLLM at `tensor-parallel-size=4`, `num_speculative_tokens=3`, and `--enforce-eager`. Greedy draft sampling (`topk=1`).
 | Category | Dataset | n | Mean Accept Length | Draft Accept Rate | Per-pos Accept Rate |
 |---|---|---:|---:|---:|---|
 ### Launch Server (vLLM)
 ```bash
+vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
   --tensor-parallel-size 4 \
   --gpu-memory-utilization 0.90 \
   --block-size 128 \
   --speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
+```