brandonmusic

/

MiniMax-M3-NVFP4

like 10

Running into K/V relayout cliff at 16K context. Short context works fine, but 16K is not exactly a world banger. So I’m not ready to release what I”m running yet, because it’s just trash for any releastic
those numbers are awful. It’s correct though. But pretty much worthless.

Once i get something that is decent speed, correct, and passes Lavd and Estonia, I’ll make a docker image and tax it here.

https://github.com/local-inference-lab/vllm/tree/dev/chthonic-consecration That will get you a good element, and install the b12x master or from this image. Luke just realeased, it, it’s v1, going about 100 t/s.

update my model card with a local image I've been running if you want to try it. @aaron-newsome

Thank you for your image. I have the problem that mm:think tags leaking in the output when i use the image verdictai/minimax-m3-nvfp4-b12x:v1. Probably because of the missing reasoning parser in the vllm Version used in the image. Do you see the same issue?

Yeah you would likely need a reasoning Parser in your runtime script. What did your script use? (I did not post one on my model card)

just used this command
docker run --gpus all --shm-size 32g
-e TP=4
-e GPU_UTIL=0.90
-e MAXLEN=256000
-e MAXSEQS=8
-e PORT=9211
-v ~/models/MiniMax-M3-NVFP4/:/model
-p 8000:9211
verdictai/minimax-m3-nvfp4-b12x:v1

that works but i get the mm:think tags in the message.
When I specify
--tool-call-parser minimax_m3
--reasoning-parser minimax_m3
--enable-auto-tool-choice

I get error the message (APIServer pid=1) KeyError: 'invalid tool call parser: minimax_m3 (chose from { apertus,....
It looks like the version in the image doesn't recognize the parser yet.

Here's my docker-compose.yaml

services:
  sglang-mm:
    image: scitrera/dgx-spark-sglang-mm:v0
    restart: unless-stopped
    container_name: minimax-m3
    entrypoint: ["/opt/nvidia/nvidia_entrypoint.sh"]
    ports:
      - "9060:8000"
    volumes:
      - /mnt/data/nvme1n1/models/MiniMax-M3-NVFP4:/models/MiniMax-M3-NVFP4:ro
    environment:
      - TORCH_NCCL_ASYNC_ERROR_HANDLING=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 4
              capabilities: [gpu]
    shm_size: "32gb"
    command: >
      sglang serve
      --model-path /models/MiniMax-M3-NVFP4
      --served-model-name minimax-m3
      --trust-remote-code
      --mem-fraction-static "0.74"
      --context-length "196608"
      --tp "4"
      --chunked-prefill-size "8192"
      --fp4-gemm-backend cutlass
      --load-format instanttensor
      --kv-cache-dtype auto
      --reasoning-parser minimax-m3
      --tool-call-parser minimax-m3
      --moe-runner-backend triton
      --trust-remote-code
      --cuda-graph-max-bs "16"
      --cuda-graph-bs "1" "2" "4" "8" "12" "16"
      --max-running-requests 16
      --allow-auto-truncate
      --host "0.0.0.0"
      --port "8000"
      --watchdog-timeout 600

it has been rock solid. around 3.5B tokens and not a single crash

awesome. I have moved onto messing with GLM 5.2.