Instructions to use cyankiwi/MiniMax-M3-AWQ-INT4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cyankiwi/MiniMax-M3-AWQ-INT4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="cyankiwi/MiniMax-M3-AWQ-INT4", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("cyankiwi/MiniMax-M3-AWQ-INT4", trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained("cyankiwi/MiniMax-M3-AWQ-INT4", trust_remote_code=True, device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cyankiwi/MiniMax-M3-AWQ-INT4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cyankiwi/MiniMax-M3-AWQ-INT4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/MiniMax-M3-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/cyankiwi/MiniMax-M3-AWQ-INT4

SGLang

How to use cyankiwi/MiniMax-M3-AWQ-INT4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cyankiwi/MiniMax-M3-AWQ-INT4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/MiniMax-M3-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cyankiwi/MiniMax-M3-AWQ-INT4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/MiniMax-M3-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use cyankiwi/MiniMax-M3-AWQ-INT4 with Docker Model Runner:
```
docker model run hf.co/cyankiwi/MiniMax-M3-AWQ-INT4
```

Get an error when performing on A800 machine

by Vigney - opened 22 days ago

Discussion

Vigney

22 days ago

[E714 14:08:26.537127252 shim_common.cpp:1458] Exception in aoti_torch: Tried to register an operator (_C::silu_and_mul_quant(Tensor($0! -> ) result, Tensor input, Tensor scale) -> ()) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /workspace/csrc/libtorch_stable/torch_bindings.cpp:10. Original registration: registered at /workspace/csrc/torch_bindings.cpp:21
vllm-minimax | Exception raised from registerDef at /pytorch/aten/src/ATen/core/dispatch/Dispatcher.cpp:248 (most recent call first):
vllm-minimax | frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x9d (0x7f94a777205d in /workspace/vllm/.venv/lib/python3.12/site-packages/torch/lib/libc10.so)
vllm-minimax | frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xc3 (0x7f94a77066fd in /workspace/vllm/.venv/lib/python3.12/site-packages/torch/lib/libc10.so)
vllm-minimax | frame #2: c10::Dispatcher::registerDef(c10::FunctionSchema, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<at::Tag, std::allocatorat::Tag >) + 0x35e (0x7f93c2a811fe in /workspace/vllm/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
vllm-minimax | frame #3: torch::Library::_def(c10::FunctionSchema&&, c10::OperatorName*, std::vector<at::Tag, std::allocatorat::Tag > const&, torch::_RegisterOrVerify) & + 0x514 (0x7f93c2ac2254 in /workspace/vllm/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
vllm-minimax | frame #4: aoti_torch_library_def + 0x99 (0x7f93c6cd45b9 in /workspace/vllm/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
vllm-minimax | frame #5: + 0x2c9e3c (0x7f929e591e3c in /workspace/vllm/vllm/_C_stable_libtorch.abi3.so)
vllm-minimax | frame #6: + 0x2cb49e (0x7f929e59349e in /workspace/vllm/vllm/_C_stable_libtorch.abi3.so)
vllm-minimax | frame #7: + 0x283c02 (0x7f929e54bc02 in /workspace/vllm/vllm/_C_stable_libtorch.abi3.so)
vllm-minimax | frame #8: + 0x647e (0x7f94a888847e in /lib64/ld-linux-x86-64.so.2)
vllm-minimax | frame #9: + 0x6568 (0x7f94a8888568 in /lib64/ld-linux-x86-64.so.2)
vllm-minimax | frame #10: _dl_catch_exception + 0xe5 (0x7f94a8692ba5 in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #11: + 0xdff6 (0x7f94a888fff6 in /lib64/ld-linux-x86-64.so.2)
vllm-minimax | frame #12: _dl_catch_exception + 0x88 (0x7f94a8692b48 in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #13: + 0xe34e (0x7f94a889034e in /lib64/ld-linux-x86-64.so.2)
vllm-minimax | frame #14: + 0x9063c (0x7f94a85ae63c in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #15: _dl_catch_exception + 0x88 (0x7f94a8692b48 in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #16: _dl_catch_error + 0x33 (0x7f94a8692c13 in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #17: + 0x9012e (0x7f94a85ae12e in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #18: dlopen + 0x48 (0x7f94a85ae6c8 in /usr/lib/x86_64-linux-gnu/libc.so.6)
vllm-minimax | frame #19: /workspace/vllm/.venv/bin/python3() [0x666d37]
vllm-minimax | frame #20: /workspace/vllm/.venv/bin/python3() [0x6658a7]
vllm-minimax | frame #21: /workspace/vllm/.venv/bin/python3() [0x56df94]
vllm-minimax | frame #22: _PyEval_EvalFrameDefault + 0x4fd7 (0x54e8b7 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #23: /workspace/vllm/.venv/bin/python3() [0x56c89d]
vllm-minimax | frame #24: PyObject_CallMethodObjArgs + 0xe1 (0x598ee1 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #25: PyImport_ImportModuleLevelObject + 0x8c4 (0x597af4 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #26: _PyEval_EvalFrameDefault + 0x6da5 (0x550685 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #27: PyEval_EvalCode + 0x99 (0x61f469 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #28: /workspace/vllm/.venv/bin/python3() [0x63dbec]
vllm-minimax | frame #29: /workspace/vllm/.venv/bin/python3() [0x56248c]
vllm-minimax | frame #30: _PyEval_EvalFrameDefault + 0x4fd7 (0x54e8b7 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #31: PyObject_CallOneArg + 0x4c (0x57bebc in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #32: /workspace/vllm/.venv/bin/python3() [0x577736]
vllm-minimax | frame #33: /workspace/vllm/.venv/bin/python3() [0x56e3f9]
vllm-minimax | frame #34: _PyEval_EvalFrameDefault + 0x1670 (0x54af50 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #35: /workspace/vllm/.venv/bin/python3() [0x56c89d]
vllm-minimax | frame #36: PyObject_CallMethodObjArgs + 0xe1 (0x598ee1 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #37: PyImport_ImportModuleLevelObject + 0x9eb (0x597c1b in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #38: _PyEval_EvalFrameDefault + 0x6da5 (0x550685 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #39: PyEval_EvalCode + 0x99 (0x61f469 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #40: /workspace/vllm/.venv/bin/python3() [0x63dbec]
vllm-minimax | frame #41: /workspace/vllm/.venv/bin/python3() [0x56248c]
vllm-minimax | frame #42: _PyEval_EvalFrameDefault + 0x4fd7 (0x54e8b7 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #43: /workspace/vllm/.venv/bin/python3() [0x56c89d]
vllm-minimax | frame #44: PyObject_CallMethodObjArgs + 0xe1 (0x598ee1 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #45: PyImport_ImportModuleLevelObject + 0x8c4 (0x597af4 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #46: _PyEval_EvalFrameDefault + 0x6da5 (0x550685 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #47: PyEval_EvalCode + 0x99 (0x61f469 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #48: /workspace/vllm/.venv/bin/python3() [0x63dbec]
vllm-minimax | frame #49: /workspace/vllm/.venv/bin/python3() [0x56248c]
vllm-minimax | frame #50: _PyEval_EvalFrameDefault + 0x4fd7 (0x54e8b7 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #51: /workspace/vllm/.venv/bin/python3() [0x56c89d]
vllm-minimax | frame #52: PyObject_CallMethodObjArgs + 0xe1 (0x598ee1 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #53: PyImport_ImportModuleLevelObject + 0x8c4 (0x597af4 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #54: _PyEval_EvalFrameDefault + 0x6da5 (0x550685 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #55: PyEval_EvalCode + 0x99 (0x61f469 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #56: /workspace/vllm/.venv/bin/python3() [0x63dbec]
vllm-minimax | frame #57: /workspace/vllm/.venv/bin/python3() [0x56248c]
vllm-minimax | frame #58: _PyEval_EvalFrameDefault + 0x4fd7 (0x54e8b7 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #59: /workspace/vllm/.venv/bin/python3() [0x56c89d]
vllm-minimax | frame #60: PyObject_CallMethodObjArgs + 0xe1 (0x598ee1 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #61: PyImport_ImportModuleLevelObject + 0x8c4 (0x597af4 in /workspace/vllm/.venv/bin/python3)
vllm-minimax | frame #62: _PyEval_EvalFrameDefault + 0x6da5 (0x550685 in /workspace/vllm/.venv/bin/python3)
vllm-minimax |
vllm-minimax | [W714 14:08:26.540479214 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
vllm-minimax | Overriding a previously registered kernel for the same operator and the same dispatch key
vllm-minimax | operator: _C::persistent_masked_m_silu_mul_quant(Tensor input, Tensor counts, Tensor($0! -> ) y_q, Tensor($1! -> ) y_s, bool use_ue8m0) -> ()
vllm-minimax | registered at /workspace/csrc/torch_bindings.cpp:21

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment