Instructions to use amd/MiniMax-M2.7-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/MiniMax-M2.7-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/MiniMax-M2.7-NVFP4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("amd/MiniMax-M2.7-NVFP4", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("amd/MiniMax-M2.7-NVFP4", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/MiniMax-M2.7-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/MiniMax-M2.7-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.7-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/amd/MiniMax-M2.7-NVFP4

SGLang

How to use amd/MiniMax-M2.7-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/MiniMax-M2.7-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.7-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/MiniMax-M2.7-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.7-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use amd/MiniMax-M2.7-NVFP4 with Docker Model Runner:
```
docker model run hf.co/amd/MiniMax-M2.7-NVFP4
```

linzhao-amd commited on Jun 3

Commit

1d7594a

verified ·

1 Parent(s): 014fc7f

Delete quark_profile.yaml

Browse files

Files changed (1) hide show

quark_profile.yaml +0 -177

quark_profile.yaml DELETED Viewed

@@ -1,177 +0,0 @@
-# Quark Profiling Results
-memory_usage:
-- step: "Start"
-  timestamp: 1778206913.5518243
-  relative_time_secs: 0.0
-  cpu_memory_mb: 4589.59
-  gpu_memory_mb: 777.22
-  disk_read_mb: 0.0
-  disk_write_mb: 0.0
-- step: "Model Loading Start"
-  timestamp: 1778206913.7514918
-  relative_time_secs: 0.19966745376586914
-  cpu_memory_mb: 4589.59
-  gpu_memory_mb: 777.22
-  disk_read_mb: 0.0
-  disk_write_mb: 0.0
-- step: "Model Loading End"
-  timestamp: 1778207378.5582051
-  relative_time_secs: 465.0063807964325
-  cpu_memory_mb: 6537.77
-  gpu_memory_mb: 63960.58
-  disk_read_mb: 436296.35
-  disk_write_mb: 0.09
-- step: "Dataset Loading Start"
-  timestamp: 1778207394.8392925
-  relative_time_secs: 481.28746819496155
-  cpu_memory_mb: 27673.3
-  gpu_memory_mb: 65073.95
-  disk_read_mb: 437001.61
-  disk_write_mb: 8369.02
-- step: "Dataset Loading End"
-  timestamp: 1778207398.4055986
-  relative_time_secs: 484.8537743091583
-  cpu_memory_mb: 27678.77
-  gpu_memory_mb: 65073.95
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Model Quantization Start"
-  timestamp: 1778207398.6073127
-  relative_time_secs: 485.0554883480072
-  cpu_memory_mb: 27678.77
-  gpu_memory_mb: 65073.95
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Model Preparation Start"
-  timestamp: 1778207398.749175
-  relative_time_secs: 485.19735074043274
-  cpu_memory_mb: 27678.77
-  gpu_memory_mb: 65073.95
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Model Preparation End"
-  timestamp: 1778207436.6778612
-  relative_time_secs: 523.1260368824005
-  cpu_memory_mb: 35183.42
-  gpu_memory_mb: 65119.96
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Advanced Algorithms Start"
-  timestamp: 1778207436.84824
-  relative_time_secs: 523.2964155673981
-  cpu_memory_mb: 35183.42
-  gpu_memory_mb: 65119.96
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Advanced Algorithms End"
-  timestamp: 1778207437.0057657
-  relative_time_secs: 523.4539413452148
-  cpu_memory_mb: 35183.42
-  gpu_memory_mb: 65119.96
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Calibration Start"
-  timestamp: 1778207437.273917
-  relative_time_secs: 523.722092628479
-  cpu_memory_mb: 35183.42
-  gpu_memory_mb: 65119.96
-  disk_read_mb: 437349.88
-  disk_write_mb: 8369.02
-- step: "Calibration End"
-  timestamp: 1778210047.9661236
-  relative_time_secs: 3134.414299249649
-  cpu_memory_mb: 36125.95
-  gpu_memory_mb: 87323.7
-  disk_read_mb: 437351.9
-  disk_write_mb: 8399.16
-- step: "Model Quantization End"
-  timestamp: 1778210176.061249
-  relative_time_secs: 3262.509424686432
-  cpu_memory_mb: 107186.82
-  gpu_memory_mb: 87275.89
-  disk_read_mb: 437352.28
-  disk_write_mb: 8399.16
-- step: "Freeze Model Start"
-  timestamp: 1778210176.2332122
-  relative_time_secs: 3262.681387901306
-  cpu_memory_mb: 107186.82
-  gpu_memory_mb: 87275.89
-  disk_read_mb: 437352.28
-  disk_write_mb: 8399.16
-- step: "Freeze Model End"
-  timestamp: 1778210190.9314125
-  relative_time_secs: 3277.3795881271362
-  cpu_memory_mb: 107521.27
-  gpu_memory_mb: 87275.89
-  disk_read_mb: 437352.28
-  disk_write_mb: 8399.16
-- step: "Export HF Safetensors Start"
-  timestamp: 1778210191.0775836
-  relative_time_secs: 3277.5257592201233
-  cpu_memory_mb: 107521.52
-  gpu_memory_mb: 87275.89
-  disk_read_mb: 437352.28
-  disk_write_mb: 8399.16
-- step: "Export HF Safetensors End"
-  timestamp: 1778210393.5681884
-  relative_time_secs: 3480.016364097595
-  cpu_memory_mb: 112519.61
-  gpu_memory_mb: 86791.54
-  disk_read_mb: 437357.31
-  disk_write_mb: 136647.9
-- step: "Model Evaluation Start"
-  timestamp: 1778210393.7849188
-  relative_time_secs: 3480.2330944538116
-  cpu_memory_mb: 112519.61
-  gpu_memory_mb: 86791.54
-  disk_read_mb: 437357.31
-  disk_write_mb: 136647.9
-- step: "Model Evaluation End"
-  timestamp: 1778214657.600004
-  relative_time_secs: 7744.048179626465
-  cpu_memory_mb: 113494.72
-  gpu_memory_mb: 87319.95
-  disk_read_mb: 437365.14
-  disk_write_mb: 136653.94
-- step: "End"
-  timestamp: 1778214657.857704
-  relative_time_secs: 7744.3058795928955
-  cpu_memory_mb: 113492.76
-  gpu_memory_mb: 87319.95
-  disk_read_mb: 437365.14
-  disk_write_mb: 136653.94
-# Summary Metrics
-total_quantization_time_seconds: 7744.3059
-peak_memory_mb: 113495.66
-peak_gpu_memory_mb: 87323.7
-total_disk_read_mb: 437365.14
-total_disk_write_mb: 136653.94
-# Metric Definitions:
-#
-# Checkpoint Metrics (per record):
-# - step: Name of the profiling checkpoint. Common steps include:
-#    - "Start": Initial state when profiling begins
-#    - "Model Loaded": After loading the ONNX model into memory
-#    - "Pre-process Start/End": Before and after model preprocessing
-#    - "Calibration Start/End": Before and after calibration data collection
-#    - "Quantization (MatMulNBits) Start/End": MatMulNBits quantization phase
-#    - "Quantization (Static) Start/End": Static quantization phase
-#    - "Post-process Start/End": Before and after post-processing
-#    - "Fast Finetune Start/End": Before and after fast finetuning (if enabled)
-# - timestamp: Unix timestamp (seconds since epoch) when this measurement was taken. Useful for correlating with external logs or events.
-# - relative_time_secs: Time elapsed (in seconds) since the "Start" step. Useful for understanding the duration of each phase relative to the beginning of profiling.
-# - cpu_memory_mb: Current Resident Set Size (RSS) in megabytes at this step. This includes memory from the main process and all child processes. RSS represents the portion of memory held in RAM (not swapped out).
-# - gpu_memory_mb: Current GPU memory usage in megabytes. This represents actual GPU memory used by the process, including allocations from PyTorch, ONNX Runtime, TensorRT, and other frameworks. Only available when PyTorch with CUDA/ROCm is installed and GPU is available.
-# - disk_read_mb: Cumulative disk bytes read (in megabytes) since the start of profiling. Measured relative to the baseline captured at the 'Start' checkpoint, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
-# - disk_write_mb: Cumulative disk bytes written (in megabytes) since the start of profiling. Measured relative to the baseline captured at the 'Start' checkpoint, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
-#
-# Summary Metrics (overall):
-# - total_quantization_time_seconds: Total elapsed time (in seconds) from the start of profiling to the end of the quantization process.
-# - peak_memory_mb: Peak resident set size (RSS) in megabytes for the main process during the entire profiling session. On Linux, this is read from VmHWM (high water mark) in /proc/<pid>/status. On Windows, this is the peak working set size. This metric may not be available on all platforms.
-# - peak_gpu_memory_mb: Peak GPU memory usage in megabytes during the entire profiling session. This is the maximum GPU memory used, including allocations from PyTorch, ONNX Runtime, TensorRT, and other frameworks. Only available when PyTorch with CUDA/ROCm is installed and GPU is available.
-# - total_disk_read_mb: Total disk bytes read (in megabytes) during the entire profiling session. Computed as the difference between the final and baseline cumulative read counters, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
-# - total_disk_write_mb: Total disk bytes written (in megabytes) during the entire profiling session. Computed as the difference between the final and baseline cumulative write counters, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
-# - peak_cache_dir_disk_usage_mb: Highest peak increase in disk usage (in megabytes) among all cache directories created during the profiling session, relative to each cache directory's size when monitoring started. Sampled every 1 second by recursively summing file sizes with os.scandir().