aquaman164

/

MiniMax-M3-VQ-2.4bit

like 0

Ok. I asked Fable to optimize the deployment of this model in my hardware (8x RTX3090). The script he made:

"#!/bin/bash
nvidia-smi -pm 1 >/dev/null 2>&1
nvidia-smi -pl 200 >/dev/null 2>&1
nvidia-smi -lgc 210,1695 >/dev/null 2>&1
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export OMP_NUM_THREADS=12
export M3VQ_CKPT=/root/models/aquaman164/MiniMax-M3-VQ-2.4bit
export VQ_CODEBOOKS_DIR=/root/vllm/vq_codebooks
export M3_TP=8
export M3_PORT=3000
export M3_SERVED=minimax-m2.7
export M3_MAXLEN=81920
export M3_GPU_UTIL=0.94
export M3_MAX_SEQS=4
export M3_EAGER=0
export M3_CUDAGRAPH_WARMUPS=2
export M3_MAX_BATCHED_TOKENS=4096

PYTHON=${VIRTUAL_ENV:+$VIRTUAL_ENV/bin/python}
PYTHON=${PYTHON:-/root/vllm/.venv/bin/python}
exec "$PYTHON" /root/vllm/vqmoe/models/minimax-m3/m3_vq_api_server.py"

With this, I was able to get 20 tokens/second. It still slow for agentic use but totally usable.