docs/vllm_deploy_guide.md

MiniMax-M2.7-MXFP416 vLLM Deployment Guide (RDNA 4 / RX 9700)

Hardware Requirements

• GPU: 8× AMD RX 9700 (RDNA 4 / gfx12xx)
• Memory: 128GB+ system RAM for 8-GPU setup
• OS: Linux with ROCm 6.x
• Docker: RDNA4-compatible vLLM image

Docker Image

The only validated runtime for this model is tcclaviger/vllm22:latest:

docker pull tcclaviger/vllm22:latest

This image includes:

• Custom Triton attention kernels tuned for RDNA4 (significantly faster than ROCm attention at long context)
• Fixed FP8 KV-cache quantization path (~2× throughput improvement)
• Pre-tuned GEMM configs for RX 9700
• MXFP4-16 kernels compiled for gfx12xx

System Setup

GPU Devices

Make sure all GPUs are visible:

rocm-smi --showid
# Should show: 0, 1, 2, 3, 4, 5, 6, 7

Power Limit (Recommended)

RDNA4 performs best with tuned power limits. Default is ~300W but 210W provides better sustained throughput on multi-GPU setups:

# Set per-GPU power limit
for i in 0 1 2 3 4 5 6 7; do
    rocm-smi --setpowerlimit $i 210
done

Note: At full power (300W) sustained speeds are lower due to thermal throttling. At 210W, sustained generation throughput is consistently higher under multi-user workloads.

Launching the Server

Single Container (8 GPUs)

docker run --name minimax-mxfp416 \
  --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  --device /dev/dri/renderD130:/dev/dri/renderD130 \
  --device /dev/dri/renderD132:/dev/dri/renderD132 \
  --device /dev/dri/renderD137:/dev/dri/renderD137 \
  --device /dev/dri/renderD138:/dev/dri/renderD138 \
  --device /dev/dri/renderD139:/dev/dri/renderD139 \
  --device /dev/dri/renderD140:/dev/dri/renderD140 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e TRUST_REMOTE_CODE=1 \
  -e PYTORCH_TUNABLEOP_ENABLED=1 \
  -e PYTORCH_TUNABLEOP_TUNING=0 \
  -e PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \
  -e GPU_MAX_HW_QUEUES=1 \
  -v /path/to/models:/app/models:ro \
  -p 8000:8000 \
  tcclaviger/vllm22:latest \
  bash -c "cp /app/models/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
    pip install -q sentencepiece && \
    exec vllm serve \
      /app/models/MiniMax-M2.7-MXFP416 \
      --served-model-name minimax-m2.7-mxfp416 \
      --host 0.0.0.0 --port 8000 \
      --trust-remote-code \
      --tensor-parallel-size 8 \
      --enable-expert-parallel \
      --disable-cascade-attn \
      --reasoning-parser minimax_m2 \
      --enable-auto-tool-choice \
      --tool-call-parser minimax_m2 \
      --enable-prefix-caching \
      --gpu-memory-utilization 0.93 \
      --max-model-len 180000 \
      --max-num-seqs 48 \
      --max-num-batched-tokens 2048 \
      --kv-cache-dtype fp8_e4m3 \
      --attention-backend TRITON_ATTN \
      --override-generation-config '{\"max_tokens\": 16384}'"

Key vLLM Flags Explained

Flag	Value	Purpose
--tensor-parallel-size	8	Split model across 8 GPUs
--enable-expert-parallel		Enable expert-parallel distribution
--disable-cascade-attn		Disable cascade attention for MoE layers
--attention-backend	TRITON_ATTN	Use Triton kernels (10× faster than ROCm on RDNA4)
--kv-cache-dtype	fp8_e4m3	FP8 KV cache (~50% memory savings)
--enable-prefix-caching		Cache common prefixes (93%+ hit rate observed)
--max-model-len	180000	180k context
--max-num-seqs	48	Max concurrent sequences
--max-num-batched-tokens	2048	Max tokens per batch
--gpu-memory-utilization	0.93	Use 93% of GPU memory

Important: The --disable-cascade-attn flag is required for MoE models. Without it, the model will produce incorrect outputs.

Running with Patches

If you have custom model patches:

-v /path/to/patches:/patches:ro \

The Docker entry point copies vllm22_minimax_m2.py to the vLLM model directory before launching. This adds MXFP4-16 support for MiniMax-M2.7.

Performance Notes

Observed Performance (8× RX 9700, 210W power limit)

• Generation throughput: 30–35 tokens/s
• Prefill throughput: 2000+ tokens/s (with prefix caching)
• Prefix cache hit rate: ~93%
• KV cache usage: 25–33% typical at 180k context
• Max concurrent users: 4–5 at full 180k context

KV Cache Capacity

With 8× RX 9700 and FP8 KV cache:

• KV cache memory: 11.35 GiB
• KV cache tokens: ~768K tokens
• Max context per request: 180,000 tokens
• Max concurrent at 180k: ~4 requests

Model Loading

• Weight loading time: ~42 seconds
• Memory per GPU (TP8): ~17.5 GiB
• Torch compile warmup: ~37 seconds

Testing the Deployment

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

completion = client.chat.completions.create(
    model="minimax-m2.7-mxfp416",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what MXFP4 quantization is in one sentence."}
    ],
    temperature=1.0,
    max_tokens=256,
)

print(completion.choices[0].message.content)

Troubleshooting

"expandable_segments not supported"

This warning is benign on ROCm. The model runs correctly despite the warning.

Low throughput at long context

Ensure TRITON_ATTN backend is active. Default ROCm attention is 10× slower on RDNA4 at long context.

Thermal throttling

If sustained throughput degrades over time, reduce power limit to 210W per GPU:

rocm-smi --setpowerlimit 0 210

Model fails to load

Ensure --trust-remote-code is set and the model path is correct. The custom model file (vllm22_minimax_m2.py) must be copied before vLLM loads the model.