Upload README.md with huggingface_hub

README.md

@@ -13,9 +13,9 @@ library_name: hipfire
 
 [hipfire](https://github.com/Kaden-Schutt/hipfire)-native quantizations of
 [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) — a 229B-parameter
-(≈10B active) Mixtral-style MoE: GQA attention with per-layer QK-norm, partial
-rotate_half RoPE, and a 256-expert top-8 sigmoid+bias router (DeepSeek-V3-style),
-SwiGLU experts, no shared expert. 62 layers, hidden 3072, vocab 200064.
+(≈10B active) MoE: GQA attention with per-layer QK-norm, partial rotate_half
+RoPE, and a 256-expert top-8 sigmoid+bias router (DeepSeek-V3-style), SwiGLU
+experts, **no shared expert**. 62 layers, hidden 3072, vocab 200064.
 
 These `.mq*` files run with the hipfire inference engine (HIP/ROCm-direct, no
 Python in the hot path). They are **not** GGUF/safetensors and won't load in
@@ -25,44 +25,58 @@ llama.cpp / transformers.
 
 | File | Expert format | Size | Target |
 |------|---------------|------|--------|
-| `MiniMax-M2.7.mq4` | MQ4G256 (4-bit FWHT) experts, Q8 attn/router/head | 123.6 GB | high quality |
-| `MiniMax-M2.7.mq2-lloyd` | MQ2G256-Lloyd (2-bit + codebook) experts, Q8 attn/router/head | ≈72 GB | fits 128 GB unified (e.g. Strix Halo / gfx1151) |
+| `MiniMax-M2.7.mq4` | MQ4G256 (4-bit FWHT) experts, Q8 attn/router/head | 123.6 GB | best quality |
+| `MiniMax-M2.7.mq3-lloyd` | MQ3G256-Lloyd (3-bit + 8-entry codebook) experts + gate-up AWQ, Q8 attn/router/head | 102.6 GB | **fits 128 GB unified (e.g. Strix Halo / gfx1151)** |
 
 ## Validation
 
 The hipfire MiniMax-M2 forward pass was validated against a PyTorch
-(`transformers`) reference on a dimension-faithful tiny random-weight oracle
-(per-layer post-residual hidden states):
+(`transformers` built-in `MiniMaxM2`) reference on a dimension-faithful tiny
+random-weight oracle (per-layer post-residual hidden-state cosine):
 
-- **mq4 experts**: cosine **0.9996** vs reference (residual = 4-bit quant noise)
-- **mq2-lloyd experts**: cosine **0.9987** vs reference (residual = 2-bit-Lloyd noise)
+- **mq4** experts: **0.9996** vs reference
+- **mq3-lloyd** experts: **0.9994**
+- attention isolated **0.99990**; routing (sigmoid + bias top-8) matches exactly.
 
-Attention isolated to cosine 0.99990; routing (sigmoid + bias top-8) matches the
-reference selection and weights exactly.
-
-The real 229B `mq4` build was confirmed end-to-end: it loads (123.6 GB) and
-generates coherent, factually-correct text (e.g. completes "The capital of
-France is" → "Paris. The capital of Germany is Berlin. ...") at ~72 tok/s decode
-on a single MI300X.
+Real 229B generation is coherent on both files (greedy): e.g. *"Explain why the
+sky is blue"* → "shorter blue wavelengths of sunlight scatter more than other
+colors … this scattered light reaches our eyes from every part of the sky";
+*"factorial of n"* → correct Python with a docstring. (Greedy decode can show
+mild repetition on open-ended prompts; use light sampling / a repeat penalty and
+the chat template for production.)
 
 ## Quantization
 
 Source: the official FP8 (E4M3 + F32 block-[128,128] `weight_scale_inv`)
 checkpoint, dequantized to F32 then re-quantized. Attention projections, router,
-embedding, and lm_head are kept at Q8; routed experts are MQ4G256 (mq4) or
-MQ2G256-Lloyd (mq2-lloyd).
+embedding, and lm_head are kept at **Q8**; routed experts are MQ4G256 (mq4) or
+MQ3G256-Lloyd (mq3-lloyd). The mq3-lloyd build bakes in **gate/up AWQ** (shared
+per-layer activation scale from an unsloth imatrix; the math `W·s @ x/s = W·x`
+is exact, redistributing quant difficulty toward the salient input channels).
+
+## Why only two formats
+
+MiniMax-M2 has **no shared expert**, so there is no higher-precision path to
+buffer routed-expert quantization error. Empirically:
+
+- **2-bit experts (mq2 / mq2-lloyd) are not shippable** — they stay grammatical
+  but collapse into repetition loops on instruction prompts (per-block 2-bit MoE
+  error reaches ~80–100% of the block's signal). AWQ alone does not rescue them.
+- **MQ3-Lloyd (3-bit) is the floor for coherent MiniMax** — 3-bit cuts the
+  per-block error to ~40–50% and recovers near-mq4 fluency, at 102.6 GB.
+- Plain (non-Lloyd) `mq3` / `mq2` are omitted because hipfire ships indexed-decode
+  MoE GEMV kernels only for MQ4G256, MQ6G256, MQ2G256-Lloyd, and MQ3G256-Lloyd.
 
 ## Usage (hipfire)
 
 ```
-hipfire serve --model MiniMax-M2.7.mq2-lloyd
+hipfire serve --model MiniMax-M2.7.mq3-lloyd
 # or a single forward:
-# examples/infer_minimax --model MiniMax-M2.7.mq4 --prompt "..."
+# examples/infer_minimax --model MiniMax-M2.7.mq3-lloyd --prompt "..."
 ```
 
 ## Notes
 
-- MQ3 / MQ2 / MQ3-Lloyd expert variants are not yet published: hipfire does not
-  currently ship indexed-decode MoE GEMV kernels for those formats (only
-  MQ4G256, MQ6G256, and MQ2G256-Lloyd have complete decode+prefill MoE kernels).
 - arch_id 10 in the hipfire HFQ header.
+- The MQ3G256-Lloyd indexed-decode MoE GEMV kernel was added in hipfire for this
+  model (first 3-bit-Lloyd MoE decode kernel in the engine).