Fixing Model Card

README.md

@@ -20,9 +20,9 @@ license: other
 
 # MiniMax-M2.5 — Quantized (compressed-tensors for vLLM)
 
-This repository contains **quantized inference builds** of **[MiniMaxAI/MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5)** exported in the **compressed-tensors** layout for **vLLM**.
+This repository contains **quantized inference builds** of **MiniMaxAI/MiniMax-M2.5** exported in the **compressed-tensors** layout for **vLLM**.
 
-MiniMax-M2.5 is a large **MoE** model (script notes: **229B params**, **256 experts** with **8 activated per token**, **62 layers**, using `block_sparse_moe` expert MLPs with `w1/w2/w3` in a SwiGLU-style structure). :contentReference[oaicite:0]{index=0}
+MiniMax-M2.5 is a large **Mixture-of-Experts (MoE)** model. The attached quant scripts calibrate **all experts** (not just router top-k) to produce more robust scales across the full mixture.
 
 ---
 
@@ -30,10 +30,10 @@ MiniMax-M2.5 is a large **MoE** model (script notes: **229B params**, **256 expe
 
 This repo publishes **two quant variants**:
 
-- **AWQ-INT4** — weight-only AWQ (INT4 weights, FP16/BF16 activations)
-- **NVFP4** — NVFP4 (FP4 weights + FP4 activations), optimized for Blackwell-class GPUs :contentReference[oaicite:1]{index=1}
+- **AWQ-INT4** — weight-only AWQ (**INT4 weights**, FP16/BF16 activations at runtime)
+- **NVFP4** — NVFP4 quant (**FP4 weights + FP4 activations**), intended for runtimes that support NVFP4 kernels
 
-> The `main` branch is typically used as a landing page; the runnable artifacts are under the variant branches above.
+> The `main` branch is typically a landing page. The runnable artifacts live under the **AWQ-INT4** and **NVFP4** branches.
 
 ---
 
@@ -45,86 +45,140 @@ Each variant branch includes:
 - `config.json` with compressed-tensors quant metadata
 - Tokenizer artifacts (and chat template assets if present)
 
-Export is done with `save_compressed=True` for vLLM compatibility. :contentReference[oaicite:2]{index=2} :contentReference[oaicite:3]{index=3}
+Exports are written with `save_compressed=True` so vLLM can load them as **compressed-tensors**.
 
 ---
 
-## Critical MoE detail: **All experts activated during calibration**
+## Critical MoE detail: all experts are activated during calibration
 
-MoE calibration is **not** performed with router top-k only. Instead, the scripts replace `MiniMaxM2SparseMoeBlock` with a calibration wrapper that runs **ALL experts** for **every sample**, ensuring reliable scale/activation statistics for every expert. :contentReference[oaicite:4]{index=4} :contentReference[oaicite:5]{index=5} :contentReference[oaicite:6]{index=6}
+Calibration is **MoE-aware**:
 
-The scripts also pass `moe_calibrate_all_experts=True` into the `oneshot(...)` call to enforce this behavior end-to-end. :contentReference[oaicite:7]{index=7} :contentReference[oaicite:8]{index=8}
+1. Each MoE block is wrapped/replaced during calibration so **ALL experts execute** for calibration forward passes.
+2. The oneshot quant call is configured to **calibrate all experts** end-to-end.
+
+**Why it matters:** If only top-k experts are exercised, rare experts can receive poor scales and quantize badly—leading to instability when those experts trigger at inference time.
 
 ---
 
-## Quantization scope: what *is* and *is not* quantized
+## Quantization scope: what is and is not quantized
 
 ### Shared rule (both variants)
-Only the **MoE expert MLP weights** are intended to be quantized:
+
+The scripts are designed to quantize **only the MoE expert MLP weights**, e.g.:
+
 - `block_sparse_moe.experts.*.w1`
 - `block_sparse_moe.experts.*.w2`
 - `block_sparse_moe.experts.*.w3`
 
-Everything else is excluded for stability (attention, routing/gate, norms, embeddings, lm_head, etc.). :contentReference[oaicite:9]{index=9} :contentReference[oaicite:10]{index=10}
+Everything else is excluded for stability (embeddings, attention, router/gate, norms, rotary, `lm_head`, etc.).
 
-### AWQ-INT4 (W4A16)
-AWQ is configured as:
-- **INT4 weights** (`num_bits=4`, `symmetric=True`)
-- **Group-wise quantization** (`strategy="group"`) with the **group size provided by CLI argument**
-- Targets: `["Linear"]`
-- Activations are not quantized (A16 runtime) :contentReference[oaicite:11]{index=11}
+---
 
-The AWQ ignore list explicitly excludes:
-- `lm_head`, embeddings
-- MoE router (`gate`, `e_score_correction_bias`)
-- attention stack (`self_attn.*`)
-- norms / rotary / MTP (if present) :contentReference[oaicite:12]{index=12}
+## AWQ-INT4 (W4A16) details
 
-AWQ smoothing/balancing mappings are set up around `post_attention_layernorm` and the expert MLP layers (`w1/w2/w3`) with `duo_scaling=True`. :contentReference[oaicite:13]{index=13}
+- **Weights:** INT4 (`num_bits=4`, symmetric)
+- **Activations:** A16 runtime (FP16/BF16)
+- **Grouping:** group-wise AWQ; group size is configured by the script/CLI
+- **Targets:** linear layers (restricted to expert MLP linears per scope)
+- **Ignored:** attention/embeddings/router/norms/`lm_head` (kept higher precision)
+- **Smoothing:** script sets up scaling maps around post-attn norms and expert MLP weights to improve stability
 
-### NVFP4
-NVFP4 is configured as:
-- `QuantizationModifier(targets="Linear", scheme="NVFP4")`
-- Ignore list excludes the same non-expert components (router, attention, norms, lm_head, etc.)
-- NVFP4 is explicitly described in-script as **FP4 weights + FP4 activations**, “per-group-16 (fixed), optimized for Blackwell.” :contentReference[oaicite:14]{index=14} :contentReference[oaicite:15]{index=15}
+---
+
+## NVFP4 details
+
+- **Weights:** FP4
+- **Activations:** FP4
+- **Targets:** linear layers (restricted to expert MLP linears per scope)
+- **Ignored:** attention/embeddings/router/norms/`lm_head`
+- **Runtime:** requires NVFP4-capable kernels (often newer GPU + software stack)
 
 ---
 
-## Calibration data, sampling, and sequence length
+## Calibration data, sample count, and sequence length
+
+Both scripts use a **dataset recipe YAML/config** that controls:
+
+- `max_seq_length`
+- shuffle + seed
+- optional `num_samples`
+- dataset sources with formatter/column mapping and per-source sample counts
 
-Both scripts load a **dataset recipe YAML** that specifies:
-- `max_seq_length` (required)
-- `shuffle` and `seed`
-- optional `num_samples` cap
-- a list of datasets (with formatter + column mapping + per-dataset sample counts) :contentReference[oaicite:16]{index=16} :contentReference[oaicite:17]{index=17}
+**Tokenization behavior**
 
-Datasets are loaded according to the YAML config, formatted into text using formatter functions (ShareGPT / prompt-answer / chat-completion / raw text), concatenated, optionally shuffled, then tokenized with:
 - `padding=False`
 - `truncation=True`
 - `max_length=MAX_SEQUENCE_LENGTH`
-- `add_special_tokens=False` :contentReference[oaicite:18]{index=18} :contentReference[oaicite:19]{index=19}
+- `add_special_tokens=False`
 
-> The exact dataset names and per-source sample counts come from your YAML recipe file. This model card intentionally describes the pipeline (and the knobs) rather than hardcoding recipe contents.
+> The exact dataset names/counts live in your recipe file; this README documents the pipeline and knobs.
 
 ---
 
-## FP8 compatibility handling (source model stored as FP8)
+## FP8 compatibility handling (base stored as FP8)
 
-The scripts load the model in **BF16** and include safeguards to:
-- convert any FP8 parameters (e.g., `float8_e4m3fn`) to BF16 for quantization compatibility
-- sanitize `quantization_config` to avoid FX-tracing serialization issues :contentReference[oaicite:20]{index=20} :contentReference[oaicite:21]{index=21}
+If the base ships FP8 parameters, the scripts:
+
+- load in BF16,
+- convert FP8 parameters to BF16 for quantization compatibility,
+- sanitize quantization-related config fields to avoid serialization/tracing issues.
 
 ---
 
 ## Quickstart (vLLM)
 
 ### AWQ-INT4 branch
-Use vLLM with compressed-tensors enabled. (Adjust TP/expert-parallel settings to your cluster.)
 
 ```bash
 pip install -U vllm
+
 vllm serve TheHouseOfTheDude/MiniMax-M2.5:AWQ-INT4 \
   --quantization compressed-tensors \
   --tensor-parallel-size 8 \
   --enable-expert-parallel \
   --dtype bfloat16
+```
+
+### NVFP4 branch
+
+```bash
+pip install -U vllm
+
+vllm serve TheHouseOfTheDude/MiniMax-M2.5:NVFP4 \
+  --quantization compressed-tensors \
+  --tensor-parallel-size 8 \
+  --enable-expert-parallel
+```
+
+**Notes**
+
+- MiniMax-M2.5 is extremely large; multi-GPU + expert parallel is strongly recommended.
+- Long context is KV-cache heavy; tune `--max-model-len`, batch size, and GPU memory utilization accordingly.
+- Serving from a local path works too—point `vllm serve` at the variant directory (e.g., `.../AWQ-INT4` or `.../NVFP4`).
+
+---
+
+## Intended use
+
+- High-throughput instruction/chat inference where MoE efficiency matters
+- Large-scale serving stacks that benefit from reduced weight bandwidth and memory
+- Long-context workloads (subject to your hardware limits)
+
+Quantization changes **weight representation only**. It does not modify tokenizer, chat template, or safety behavior. Apply your own safety policies/filters as appropriate.
+
+---
+
+## Lineage
+
+- **Base model:** https://huggingface.co/MiniMaxAI/MiniMax-M2.5
+- **This repo:** quantized inference variants exported to **compressed-tensors** for vLLM:
+  - **AWQ-INT4**
+  - **NVFP4**
+
+---
+
+## Changelog
+
+- **v1 (current)** — Initial release with two quant variants:
+  - **AWQ-INT4** (expert-only W4A16 AWQ; all-experts calibration; group size configurable in script)
+  - **NVFP4** (FP4 weights + FP4 activations; expert-only scope; all-experts calibration; requires NVFP4-capable runtime)