brandonmusic

/

MiniMax-M3-NVFP4

like 10

Update: I was able to start the model with this workaround:

sed -i "s/--kv-cache-dtype auto/--kv-cache-dtype bfloat16/g" /opt/m3/entrypoint.sh /opt/m3/serving/boot_m3.sh

The full command works with:

TP=4
MAXLEN=196000
GPU_UTIL=0.93
MAXSEQS=4

However, I’m not sure whether this workaround preserves the intended model quality. Since the image defaults to --kv-cache-dtype auto, and auto resolves to fp8_e4m3, forcing bfloat16 may change the intended runtime behavior.

Could you confirm whether using --kv-cache-dtype bfloat16 is the correct/validated configuration for this quant, or whether it may affect output quality compared to your expected setup?

There are updated suggested images, on github "local inference lab" for rtxpro6k. Or the other comment where we discussd simlar issue: services:
sglang-mm:
image: scitrera/dgx-spark-sglang-mm:v0
restart: unless-stopped
container_name: minimax-m3
entrypoint: ["/opt/nvidia/nvidia_entrypoint.sh"]
ports:
- "9060:8000"
volumes:
- /mnt/data/nvme1n1/models/MiniMax-M3-NVFP4:/models/MiniMax-M3-NVFP4:ro
environment:
- TORCH_NCCL_ASYNC_ERROR_HANDLING=1
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 4
capabilities: [gpu]
shm_size: "32gb"
command: >
sglang serve
--model-path /models/MiniMax-M3-NVFP4
--served-model-name minimax-m3
--trust-remote-code
--mem-fraction-static "0.74"
--context-length "196608"
--tp "4"
--chunked-prefill-size "8192"
--fp4-gemm-backend cutlass
--load-format instanttensor
--kv-cache-dtype auto
--reasoning-parser minimax-m3
--tool-call-parser minimax-m3
--moe-runner-backend triton
--trust-remote-code
--cuda-graph-max-bs "16"
--cuda-graph-bs "1" "2" "4" "8" "12" "16"
--max-running-requests 16
--allow-auto-truncate
--host "0.0.0.0"
--port "8000"
--watchdog-timeout 600

b12x works with fp8 cache just fine.

Update: I was able to start the model with this workaround:

sed -i "s/--kv-cache-dtype auto/--kv-cache-dtype bfloat16/g" /opt/m3/entrypoint.sh /opt/m3/serving/boot_m3.sh

Hi, thanks for this fix ! Another problem, am I alone to have a problem with tools calling ? No minimax_m3 tools available, and minimax_m2 does not work.

@Jonah16 do you have --reasoning-parser minimax_m3
--tool-call-parser minimax_m3 --enable-auto-tool-choice in your launch script?
In the other community thread i linked to a working image.

@brandonmusic (APIServer pid=1) KeyError: 'invalid tool call parser: minimax_m3 (chose from { apertus,cohere_command3,cohere_command4,deepseek_v3,deepseek_v31,deepseek_v32,deepseek_v4,ernie45,functiongemma,gemma4,gigachat3,glm45,glm47,granite,granite-20b-fc,granite4,hermes,hunyuan_a13b,hy_v3,internlm,jamba,kimi_k2,lfm2,llama3_json,llama4_json,llama4_pythonic,longcat,mimo,minicpm5,minimax,minimax_m2,mistral,olmo3,openai,phi4_mini_json,poolside_v1,pythonic,qwen3_coder,qwen3_xml,seed_oss,step3,step3p5,xlam })'

I think my vllm version is too old. I am checking if its the problem.

yeah rebase. or upgrade it. --tool-call-parser minimax_m3
--reasoning-parser minimax_m3
--enable-auto-tool-choice pulled that directly fro vllm's cookbook for the model.

@brandonmusic i also had a problem with your model on vllm, so switched to the sglang container you have mentioned. But i couldnt make it work with quantisized kv_cache. Does your current image support something exept fp16 ?

yeah it was with fp8 cache. check the other respondes, linked to some images taht might help you, if you are having issues.