First README.md

README.md

@@ -0,0 +1,130 @@
+---
+language:
+- en
+library_name: vllm
+pipeline_tag: text-generation
+tags:
+  - text-generation
+  - conversational
+  - moe
+  - quantized
+  - compressed-tensors
+  - awq
+  - w4a16
+  - nvfp4
+base_model: MiniMaxAI/MiniMax-M2.5
+base_model_relation: quantized
+quantized_by: TheHouseOfTheDude
+license: other
+---
+
+# MiniMax-M2.5 — Quantized (compressed-tensors for vLLM)
+
+This repository contains **quantized inference builds** of **[MiniMaxAI/MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5)** exported in the **compressed-tensors** layout for **vLLM**.
+
+MiniMax-M2.5 is a large **MoE** model (script notes: **229B params**, **256 experts** with **8 activated per token**, **62 layers**, using `block_sparse_moe` expert MLPs with `w1/w2/w3` in a SwiGLU-style structure). :contentReference[oaicite:0]{index=0}
+
+---
+
+## Variants / Branches
+
+This repo publishes **two quant variants**:
+
+- **AWQ-INT4** — weight-only AWQ (INT4 weights, FP16/BF16 activations)
+- **NVFP4** — NVFP4 (FP4 weights + FP4 activations), optimized for Blackwell-class GPUs :contentReference[oaicite:1]{index=1}
+
+> The `main` branch is typically used as a landing page; the runnable artifacts are under the variant branches above.
+
+---
+
+## What’s inside (per variant)
+
+Each variant branch includes:
+
+- Sharded quantized weights (`*.safetensors`) + `model.safetensors.index.json`
+- `config.json` with compressed-tensors quant metadata
+- Tokenizer artifacts (and chat template assets if present)
+
+Export is done with `save_compressed=True` for vLLM compatibility. :contentReference[oaicite:2]{index=2} :contentReference[oaicite:3]{index=3}
+
+---
+
+## Critical MoE detail: **All experts activated during calibration**
+
+MoE calibration is **not** performed with router top-k only. Instead, the scripts replace `MiniMaxM2SparseMoeBlock` with a calibration wrapper that runs **ALL experts** for **every sample**, ensuring reliable scale/activation statistics for every expert. :contentReference[oaicite:4]{index=4} :contentReference[oaicite:5]{index=5} :contentReference[oaicite:6]{index=6}
+
+The scripts also pass `moe_calibrate_all_experts=True` into the `oneshot(...)` call to enforce this behavior end-to-end. :contentReference[oaicite:7]{index=7} :contentReference[oaicite:8]{index=8}
+
+---
+
+## Quantization scope: what *is* and *is not* quantized
+
+### Shared rule (both variants)
+Only the **MoE expert MLP weights** are intended to be quantized:
+- `block_sparse_moe.experts.*.w1`
+- `block_sparse_moe.experts.*.w2`
+- `block_sparse_moe.experts.*.w3`
+
+Everything else is excluded for stability (attention, routing/gate, norms, embeddings, lm_head, etc.). :contentReference[oaicite:9]{index=9} :contentReference[oaicite:10]{index=10}
+
+### AWQ-INT4 (W4A16)
+AWQ is configured as:
+- **INT4 weights** (`num_bits=4`, `symmetric=True`)
+- **Group-wise quantization** (`strategy="group"`) with the **group size provided by CLI argument**
+- Targets: `["Linear"]`
+- Activations are not quantized (A16 runtime) :contentReference[oaicite:11]{index=11}
+
+The AWQ ignore list explicitly excludes:
+- `lm_head`, embeddings
+- MoE router (`gate`, `e_score_correction_bias`)
+- attention stack (`self_attn.*`)
+- norms / rotary / MTP (if present) :contentReference[oaicite:12]{index=12}
+
+AWQ smoothing/balancing mappings are set up around `post_attention_layernorm` and the expert MLP layers (`w1/w2/w3`) with `duo_scaling=True`. :contentReference[oaicite:13]{index=13}
+
+### NVFP4
+NVFP4 is configured as:
+- `QuantizationModifier(targets="Linear", scheme="NVFP4")`
+- Ignore list excludes the same non-expert components (router, attention, norms, lm_head, etc.)
+- NVFP4 is explicitly described in-script as **FP4 weights + FP4 activations**, “per-group-16 (fixed), optimized for Blackwell.” :contentReference[oaicite:14]{index=14} :contentReference[oaicite:15]{index=15}
+
+---
+
+## Calibration data, sampling, and sequence length
+
+Both scripts load a **dataset recipe YAML** that specifies:
+- `max_seq_length` (required)
+- `shuffle` and `seed`
+- optional `num_samples` cap
+- a list of datasets (with formatter + column mapping + per-dataset sample counts) :contentReference[oaicite:16]{index=16} :contentReference[oaicite:17]{index=17}
+
+Datasets are loaded according to the YAML config, formatted into text using formatter functions (ShareGPT / prompt-answer / chat-completion / raw text), concatenated, optionally shuffled, then tokenized with:
+- `padding=False`
+- `truncation=True`
+- `max_length=MAX_SEQUENCE_LENGTH`
+- `add_special_tokens=False` :contentReference[oaicite:18]{index=18} :contentReference[oaicite:19]{index=19}
+
+> The exact dataset names and per-source sample counts come from your YAML recipe file. This model card intentionally describes the pipeline (and the knobs) rather than hardcoding recipe contents.
+
+---
+
+## FP8 compatibility handling (source model stored as FP8)
+
+The scripts load the model in **BF16** and include safeguards to:
+- convert any FP8 parameters (e.g., `float8_e4m3fn`) to BF16 for quantization compatibility
+- sanitize `quantization_config` to avoid FX-tracing serialization issues :contentReference[oaicite:20]{index=20} :contentReference[oaicite:21]{index=21}
+
+---
+
+## Quickstart (vLLM)
+
+### AWQ-INT4 branch
+Use vLLM with compressed-tensors enabled. (Adjust TP/expert-parallel settings to your cluster.)
+
+```bash
+pip install -U vllm
+vllm serve TheHouseOfTheDude/MiniMax-M2.5:AWQ-INT4 \
+  --quantization compressed-tensors \
+  --tensor-parallel-size 8 \
+  --enable-expert-parallel \
+  --dtype bfloat16