Text Generation
Transformers
Safetensors
minimax_m2
neuralmagic
redhat
llmcompressor
quantized
INT8
conversational
custom_code
8-bit precision
compressed-tensors
Instructions to use RedHatAI/MiniMax-M2.5-quantized.w8a8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RedHatAI/MiniMax-M2.5-quantized.w8a8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="RedHatAI/MiniMax-M2.5-quantized.w8a8", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("RedHatAI/MiniMax-M2.5-quantized.w8a8", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("RedHatAI/MiniMax-M2.5-quantized.w8a8", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use RedHatAI/MiniMax-M2.5-quantized.w8a8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RedHatAI/MiniMax-M2.5-quantized.w8a8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/MiniMax-M2.5-quantized.w8a8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/RedHatAI/MiniMax-M2.5-quantized.w8a8
- SGLang
How to use RedHatAI/MiniMax-M2.5-quantized.w8a8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "RedHatAI/MiniMax-M2.5-quantized.w8a8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/MiniMax-M2.5-quantized.w8a8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "RedHatAI/MiniMax-M2.5-quantized.w8a8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/MiniMax-M2.5-quantized.w8a8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use RedHatAI/MiniMax-M2.5-quantized.w8a8 with Docker Model Runner:
docker model run hf.co/RedHatAI/MiniMax-M2.5-quantized.w8a8
File size: 1,464 Bytes
7a95a38 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | {
"</minimax:tool_call>": 200053,
"</think>": 200051,
"<add_file>": 200036,
"<code_context>": 200043,
"<code_interpreter>": 200023,
"<commit_after>": 200018,
"<commit_before>": 200016,
"<commit_message>": 200040,
"<commit_msg>": 200017,
"<delete_file>": 200037,
"<edit_file>": 200039,
"<empty_output>": 200015,
"<empty_source_file>": 200041,
"<file_content>": 200044,
"<file_sep>": 200049,
"<filename>": 200006,
"<filepath>": 200048,
"<fim_middle>": 200002,
"<fim_pad>": 200004,
"<fim_prefix>": 200001,
"<fim_suffix>": 200003,
"<function_call>": 200022,
"<gh_stars>": 200007,
"<issue_closed>": 200010,
"<issue_comment>": 200009,
"<issue_start>": 200008,
"<jupyter_code>": 200013,
"<jupyter_error>": 200035,
"<jupyter_output>": 200014,
"<jupyter_start>": 200011,
"<jupyter_text>": 200012,
"<minimax:tool_call>": 200052,
"<pr_start>": 200046,
"<rename_file>": 200038,
"<repo_struct>": 200042,
"<reponame>": 200005,
"<review_comment>": 200047,
"<source_files>": 200045,
"<think>": 200050,
"[e~[": 200020,
"]!d~[": 200021,
"]!p~[": 200000,
"]<]end of image[>[": 200030,
"]<]end of speech[>[": 200028,
"]<]end of video[>[": 200032,
"]<]image[>[": 200025,
"]<]speech[>[": 200024,
"]<]start of image[>[": 200029,
"]<]start of speech[>[": 200027,
"]<]start of video[>[": 200031,
"]<]video[>[": 200026,
"]<]vision pad[>[": 200033,
"]~!b[": 200034,
"]~b]": 200019
}
|