README.md

---
base_model:
  - MiniMaxAI/MiniMax-M3
base_model_relation: finetune
license: other
license_name: minimax-model-license
license_link: https://huggingface.co/MiniMaxAI/MiniMax-M3
language:
  - en
pipeline_tag: text-generation
library_name: vllm
tags:
  - minimax
  - minimax-m3
  - moe
  - mixture-of-experts
  - reap
  - pruned
  - quantized
  - rfi
  - code
  - rocm
  - gfx1201
  - rdna4
---

> # ⚠️ Vision is not yet working — text-only for now.
> Image/video input is **not functional** in this build. The MiniMax-M3 vision
> tower has an unresolved rotary-embedding bug, so multimodal input is disabled
> in the serving config. **Text generation works**; treat this as a text-only
> coder model until vision is fixed.

# MiniMax-M3-Coder (RFI)

A REAP-pruned, RFI-quantized **coder** build of MiniMax-M3, packaged to serve on
AMD ROCm / RDNA4 (gfx1201) through a custom vLLM fork.

> ### ⚠️ Experimental — in active development
> This model is **functional but experimental**. It runs and produces coherent
> long-context output, but it has not been broadly evaluated, the quantization
> recipe is still being tuned, and behavior may change between revisions. Do not
> rely on it for production workloads. Use at your own risk.

---

## What this is

`MiniMax-M3-Coder` is a derivative of **[MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3)**:

1. **Expert pruning (REAP).** The set of routed experts to drop was taken from the
   expert inventory of
   **[JANGQ-AI/MiniMax-M3-REAP22-Coder](https://huggingface.co/JANGQ-AI/MiniMax-M3-REAP22-Coder)**.
   The surviving experts are renumbered contiguously and the router gate/bias are
   sliced to match. This build carries **100 routed experts** (top-4) plus 1
   shared expert.
2. **Mixed-precision RFI quantization.** A composite, mostly-low-bit scheme
   (details below) whose *mixed format* was inspired by the REAP22-Coder model.
3. **Runtime.** Serving support (custom RFI quant kernels + MiniMax-M3 sparse
   "lightning indexer" attention) lives in the `tcclaviger/vllm22` vLLM fork,
   published as the **`:dev`** image.

### Sources & credits

| Role | Source |
| --- | --- |
| Original base model (we are a derivative / fine-tune of it) | [MiniMaxAI/MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) |
| Expert-prune inventory (which experts to drop) + mixed-quant inspiration | [JANGQ-AI/MiniMax-M3-REAP22-Coder](https://huggingface.co/JANGQ-AI/MiniMax-M3-REAP22-Coder) |
| Concept inspiration (the idea of an M3 Coder build) | **JANG2_L (vMLX)** |
| Pruning + RFI quantization + vLLM serving fork | [tcclaviger](https://huggingface.co/tcclaviger) |

All rights, the license, and the underlying model belong to **MiniMaxAI**; this
repository is a derivative work and inherits the
[MiniMax-M3 license](https://huggingface.co/MiniMaxAI/MiniMax-M3). Thanks to
MiniMaxAI for the base model, to JANGQ-AI for the REAP expert inventory and the
mixed-quant direction, and to JANG2_L (vMLX) for the original M3-Coder mixed precision idea.

---

## Architecture (post-prune)

| Field | Value |
| --- | --- |
| Backbone | MiniMax-M3 sparse MoE (DSA "lightning indexer" attention) |
| Hidden size | 6144 |
| Layers | 60 |
| Attention heads / KV heads | 64 / 4 (head dim 128, partial RoPE 0.5, θ=5e6) |
| Routed experts (post-REAP) | 100, top-4 |
| Shared experts | 1 |
| Expert intermediate size | 3072 |
| Sparse indexer | index dim 128, 4 index heads, top-k 16 blocks × 128 |
| Vocab | 200,064 |
| Max context | 1,048,576 (1M) |
| Multimodal | vision tower present in weights, **not yet functional** (rotary bug) — disabled in the serving config |

---

## Quantization schema — RFI composite v1.0

**RFI** is a rotation-based composite integer scheme (`quant_method: rfi`,
`format: rfi-quantized`, `version: rfi-composite-1.0`, status `compressed`).

**Global settings (all quantized Linears):**
- **Rotation:** FWHT-32 (`hadamard32`) applied before quantization.
- **Granularity:** per-group, `group_size = 32`, **symmetric**.
- **Scales:** block-float, int8 mantissa + int8 exponent
  (`scale_encoding: block_float_i8m_i8e`).
- Target: `Linear` layers.

**Per-component bit widths (mixed):**

| Component | Bits | Notes |
| --- | --- | --- |
| Routed MoE experts (`block_sparse_moe.experts`) | **2-bit** | codebook `{-10, -3, 3, 10}` — the bulk of the weights |
| Shared expert (`shared_experts`) | **6-bit** | |
| Vision tower / multimodal projector / patch-merge MLP | **6-bit** | |
| Everything else (attention q/k/v/o, dense MLP, …) | **8-bit** | int, default group |

**Kept unquantized (fp16 / bf16 — in the `ignore` list):**
- Router gates: `block_sparse_moe.gate` (per-layer) and `gate.weight`
- Token embeddings: `model.embed_tokens`
- Output head: `lm_head`
- **Lightning-indexer projections: `index_q_proj`, `index_k_proj`**

---

## Running it

Serving requires the custom fork image (stock vLLM does **not** include the RFI
quant kernels or the MiniMax-M3 sparse-attention path):

- **Docker image:** [`tcclaviger/vllm22`](https://hub.docker.com/repository/docker/tcclaviger/vllm22/general) — use the **`:dev`** tag.

```bash
docker pull tcclaviger/vllm22:dev
```

### Reference hardware: 4× AMD Radeon AI Pro R9700

This model runs on **4× AMD AI Pro R9700** (gfx1201 / RDNA4, 32 GB each;
ROCm 7.2.1) at **tensor-parallel 4**, weighing in at **~106.9 GB total / ~26.7 GB
per GPU** (~2.53 bpw). It runs with these **KV-cache type and memory/space
parameters** — most importantly:

| Parameter | Value | Why |
| --- | --- | --- |
| `--kv-cache-dtype` | `fp8` | halves KV-cache footprint — the "kv type" |
| `--gpu-memory-utilization` | `0.94` | the per-GPU "space" budget |
| `--max-model-len` | `150000` | usable context within that budget |
| `--max-num-batched-tokens` | `1024` | chunked-prefill chunk size |
| `--max-num-seqs` | `1` | single-stream (KV-space bound) |
| `--block-size` | `128` | matches the M3 sparse-attention / indexer page size |
| `--tensor-parallel-size` | `4` | one shard per R9700 |

Equivalent `docker run` invocation:

```bash
docker run --rm \
  --device /dev/kfd --device /dev/dri --group-add video \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  --ipc host --network host --shm-size 32g \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -v /path/to/MiniMax-M3-Coder:/app/models \
  tcclaviger/vllm22:dev \
  /app/models \
    --served-model-name M3-Coder \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.94 \
    --max-model-len 150000 \
    --kv-cache-dtype fp8 \
    --block-size 128 \
    --max-num-seqs 1 \
    --max-num-batched-tokens 1024 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --chat-template /app/models/chat_template.jinja \
    --reasoning-parser minimax_m3 \
    --tool-call-parser minimax_m3 --enable-auto-tool-choice \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --host 0.0.0.0 --port 8078
```

`--quantization` is auto-detected from the checkpoint (`quant_method: rfi`). The
sparse lightning-indexer, RFI dequant, and mixed-bit experts are handled
automatically by the image. `HIP_VISIBLE_DEVICES=0,1,2,3` pins the four R9700s
and excludes the Raphael iGPU.

## Speculative decoding — bundled EAGLE3 draft (`eagle3/`)

This repo bundles an **RFI6-quantized EAGLE3 draft** under [`eagle3/`](./eagle3)
for roughly **2× faster decoding** via speculative decoding. It is the
[Inferact/MiniMax-M3-EAGLE3](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3)
drafter (a 1-layer head trained with TorchSpec on M3-regenerated code/math),
here quantized to **RFI6** — 6-bit rotation-int RFI on attention + MLP + the
EAGLE3 `fc` fusion, with embeddings / lm_head / norms kept BF16 (5.5 GB). At
serve time it shares this model's embedding + LM head.

Enable it by adding `--speculative-config` pointing at the subfolder (the draft
mounts with the model at `/app/models`):

```bash
  # ...same serve flags as above...
  --speculative-config '{"method": "eagle3", "model": "/app/models/eagle3", "num_speculative_tokens": 4}'
```

Speculative decoding is **lossless** — the target verifies every drafted token,
so output is byte-identical to running without the draft; only speed changes.

**Measured on this stack** (4× R9700, `num_speculative_tokens=4`) — *mean accepted
length* is tokens emitted per target forward pass (the direct speedup proxy):

| Workload | Draft accept rate | Mean accepted length | Per-position accept (pos 1–4) |
| --- | --- | --- | --- |
| Math / reasoning (low temp) | ~76% | **~4.0 tokens/step** | 0.91 / 0.80 / 0.71 / 0.61 |
| Coding (low temp) | ~53% | **~3.1 tokens/step** | 0.78 / 0.58 / 0.44 / 0.33 |
| Agentic chat (sampled) | ~36% | **~2.4 tokens/step** | 0.64 / 0.40 / 0.25 / 0.14 |

Math/reasoning is the most predictable (near the `k+1=5` ceiling), then coding,
then sampled agentic chat; net **~2.4–4× fewer target forward passes**, i.e. the
observed **>2× decode speedup**.
(Measured on the more-aggressively-pruned 87-expert "Lite" build; this 100-expert
model should match or slightly exceed these.) The draft was trained against the
full 128-expert MXFP8 M3, so acceptance on a REAP-pruned RFI target runs below the
drafter's published numbers but is still a clear win. Requires
`tcclaviger/vllm22:dev`, whose RFI runtime loads the RFI6 draft directly. Credit:
original EAGLE3 drafter by [Inferact](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3)
(TorchSpec); RFI6 quantization by tcclaviger.

## Server-side features (tooling & thinking)

These are handled **at the server/model level** — no client-side changes are
required:

- **Custom Python tool-call parser.** Tool calling uses a **Python**
  tool-call parser (`--tool-call-parser minimax_m3` with
  `--enable-auto-tool-choice`) written for this model. It runs on vLLM's Python
  parsing path — the Rust (`vllm-rs`) tool-parser components are **not** used.
- **Qwen-style thinking toggle.** Reasoning is enabled/disabled with a simple
  boolean, Qwen-style: `--default-chat-template-kwargs '{"enable_thinking": true}'`
  (or `false`), and per-request via the same `enable_thinking` chat-template
  kwarg. The chat template and the reasoning parser were modified to map this
  boolean to the model's thinking mode.
- **Guarded thinking tags (both families).** The chat template and reasoning
  parser are guarded so that **both** `<think>…</think>` **and**
  `<mm:think>…</mm:think>` are accepted, normalized, and resolved correctly —
  including in replayed conversation history — entirely on the server. Clients
  can send either tag family (or none) without any special handling.

---

## Disclaimer

This is an **experimental, in-development** research artifact provided as-is, with
no warranty. It is a derivative of MiniMax-M3 and is subject to the upstream
MiniMax-M3 license and usage terms. Evaluate quality and safety yourself before
any use.