rocmfpx-quantization-guide.md

# ROCmFPX Quantization Guide — q4_0_rocmfp4_fast experts + q8_0_rocmfpx everything else

Self-contained playbook for producing "hybrid" ROCmFPX GGUF quants from a Hugging Face repo link:
**routed MoE experts (up/down/gate) → `q4_0_rocmfp4_fast`, every other tensor → `q8_0_rocmfpx`**.
Runs fully **CPU-only** (no GPU needed; quantization is a CPU reference path).

Validated end-to-end on:
| Model | Source | Recipe applied to | Result |
|---|---|---|---|
| Laguna-S-2.1 (118B-A10B, poolside, arch `laguna`) | `unsloth/Laguna-S-2.1-GGUF/BF16` (5 shards) | `ffn_{gate,up,down}_exps` | 224 GB BF16 → 61.6 GB (4.39 bpw) |
| Qwen3.8-27B (dense) | `unsloth/Qwen3.8-27B-GGUF/BF16` (2 shards) | n/a (dense) | 52 GB BF16 → 26.9 GB (8.25 bpw) |
| G4-MeroMero-26B-A4B (Gemma4 MoE) | raw HF weights `llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic` | `ffn_gate_up_exps`, `ffn_down_exps` | converted to BF16 GGUF (50.5 GB) → 14.0 GB (4.64 bpw) |
| Qwen3.8-27B hybrid (dense, `qwen35` arch) | `unsloth/Qwen3.8-27B-GGUF/BF16` | sensitive tensors per Unsloth UD-Q4_K_XL (see §8) | 16.4 GB (5.15 bpw), targets ~16 GB VRAM-fit file |

---

## 0. One-time setup (CPU-only)

```bash
# System deps: gcc, cmake, git, python3 + pip
git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX

# CPU-only build (no HIP/CUDA/Vulkan/Metal)
cmake -B build-cpu -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
      -DGGML_METAL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build-cpu -j "$(nproc)" --target llama-quantize llama-gguf-split llama-gguf

# Python deps for HF→GGUF conversion (only needed for Workflow B)
pip install -r requirements/requirements-convert_hf_to_gguf.txt
pip install -e gguf-py          # install the repo's gguf lib (older pip gguf can't read ROCmFPX types!)
```

Binaries land in `build-cpu/bin/`: `llama-quantize`, `llama-gguf-split`, `llama-gguf`.

> **Note:** the ROCmFPX quant types (`q4_0_rocmfp4_fast` = 101, `q8_0_rocmfpx` = 103, etc.)
> are only understood by this fork's tooling and recent-ish llama.cpp. Use the binaries built here.

---

## 1. Decide which workflow applies

Look at the HF repo (`https://huggingface.co/<repo>`):

- **Workflow A** — repo already contains GGUF files, e.g. `BF16/*-00001-of-0000N.gguf`
  (typical for `unsloth/*-GGUF` repos). Skip straight to quantizing.
- **Workflow B** — repo contains raw weights (`model*.safetensors` + `config.json`).
  Convert to BF16 GGUF first, then quantize.

Check sizes before downloading:

```bash
curl -s https://huggingface.co/api/models/<ORG>/<REPO>/tree/main/BF16 | \
  python3 -c "import json,sys; [print(s['path'], round(s['size']/1e9,2),'GB') for s in json.load(sys.stdin)]"
```

Download with `hf` CLI (shards download in parallel; pass the first shard to llama.cpp later —
remaining shards are discovered automatically):

```bash
pip install -q huggingface_hub
hf download <ORG>/<REPO> --repo-type model --include "BF16/*" --local-dir /workspace/models/<NAME>
```

---

## 2. Find the routed-expert tensor names (critical step)

Tensor naming depends on architecture. Inspect the first GGUF shard (header only — works as soon as
the file starts existing, and only needs a few MB):

```bash
python3 - <<'EOF'
from gguf import GGUFReader
import re, collections
r = GGUFReader("<first-shard>.gguf", "r")
print("arch:", bytes(r.fields['general.architecture'].parts[-1]).decode())
names = sorted(set(re.sub(r'^blk\.\d+', 'blk.N', t.name) for t in r.tensors))
for n in names: print(n)
print(collections.Counter(t.tensor_type.name for t in r.tensors))
EOF
```

Build the expert regex from the patterns you see:

| What you see in the tensor list | Expert regex for `--tensor-type` |
|---|---|
| `blk.N.ffn_gate_exps.weight`, `ffn_up_exps`, `ffn_down_exps` (packed, e.g. `laguna`) | `ffn_(gate\|up\|down)_exps` |
| `blk.N.ffn_gate_up_exps.weight`, `ffn_down_exps` (fused gate+up, e.g. `gemma4`) | `ffn_(gate_up\|down)_exps\.weight` |
| `blk.N.ffn_gate.M`, `ffn_up.M`, `ffn_down.M` (per-expert tensors) | `ffn_(gate\|up\|down)\.\d+\.` |

Patterns are **regexes matched against lowercased tensor names** (regex_search, i.e. partial match).
Norms, biases, router scales (`*.scale`, `*.bias`, 1-D tensors) are never quantized regardless, so
matching a little too broadly is harmless — but still verify with a dry run.

---

## 3. Dry run (always do this first)

`--output-tensor-type` only affects `output.weight`; to get "everything else = q8_0_rocmfpx" use the
**base ftype `Q8_0_ROCMFPX`** and override experts:

```bash
./build-cpu/bin/llama-quantize --dry-run \
  --tensor-type "<EXPERT_REGEX>=q4_0_rocmfp4_fast" \
  <input-BF16-shard-1.gguf> <output-name.gguf> Q8_0_ROCMFPX
```

Confirm in the output:
- `applying manual override: q8_0_rocmfp4_fast...` lines appear for **every** expert tensor
  (count = layers × {gate,up,down}, e.g. 47 MoE layers × 3 = 141 for Laguna),
- attention / shared experts / embeddings → `(q8_0_rocmfpx)`,
- norms/biases/routers stay `f32`,
- the final `quant size = ... (X.XX BPW)` line looks sane (~4.4 bpw for mostly-expert MoE,
  ~8.25 bpw for dense).

For a dense model, drop `--tensor-type` entirely.

---

## 4. Quantize

```bash
./build-cpu/bin/llama-quantize \
  --tensor-type "<EXPERT_REGEX>=q4_0_rocmfp4_fast" \
  <input-BF16-shard-1.gguf> <output-name.gguf> Q8_0_ROCMFPX
```

Flags worth knowing:
- `--keep-split` — output keeps the input's shard layout.
- Default (no flag) — auto-splits output at 50 GiB.
- `--tensor-type` can be repeated, or use `--tensor-type-file list.txt` (one `name=type` per line).
- Add trailing `nthreads` positional arg to cap thread count.

Speed reference (64-core server): 118B MoE ≈ 8 min, 27B dense ≈ 1 min, 26B MoE ≈ 3 min.
Memory use is modest (mmap-based); RAM ≈ source size is plenty.

---

## 5. Verify + merge shards

Verify the result (works on merged or shard-1 file):

```bash
./build-cpu/bin/llama-quantize --dry-run <output.gguf> /tmp/dummy.gguf COPY | grep "type .*tensors"
# e.g.:  f32: 287 | q4_0_rocmfp4_fast: 141 | q8_0_rocmfpx: 386
```

Merge shards into one file (only pass the first shard; the rest are auto-discovered):

```bash
./build-cpu/bin/llama-gguf-split --merge \
  <output>-00001-of-0000N.gguf <output-merged.gguf>
```

(`--split` / `--split-max-size` do the reverse. Note: the `llama-gguf <file> r` example tool aborts
on ROCmFPX types even for valid files — use the quantize-dry-run check above instead.)

---

## 6. Workflow B only: HF weights → BF16 GGUF

For repos with `*.safetensors` + `config.json` (check `architectures` in config.json is registered in
`convert_hf_to_gguf.py` — e.g. `Gemma4ForConditionalGeneration` is supported by this fork):

```bash
hf download <ORG>/<REPO> --local-dir /workspace/models/<NAME>   # all files incl. tokenizer

python3 convert_hf_to_gguf.py /workspace/models/<NAME> \
  --outfile /workspace/models/<NAME>-BF16.gguf --outtype bf16
# for multimodal models, add --mmproj to also export the vision projector (separate file)
```

Then continue at **Step 2** with the produced BF16 GGUF.

---

## 7. Cheat sheet of the runs documented here

```bash
# Laguna-S-2.1 (laguna arch, packed experts)
./build-cpu/bin/llama-quantize \
  --tensor-type "ffn_(gate|up|down)_exps=q4_0_rocmfp4_fast" \
  --keep-split \
  Laguna-S-2.1-BF16-00001-of-00005.gguf Laguna-S-2.1-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX

# Qwen3.8-27B (dense — pure q8_0_rocmfpx)
./build-cpu/bin/llama-quantize \
  Qwen3.8-27B-BF16-00001-of-00002.gguf Qwen3.8-27B-Q8_0_ROCMFPX.gguf Q8_0_ROCMFPX

# Gemma4 MoE (fused gate_up experts), after convert_hf_to_gguf.py
./build-cpu/bin/llama-quantize \
  --tensor-type "ffn_(gate_up|down)_exps.weight=q4_0_rocmfp4_fast" \
  G4-MeroMero-26B-A4B-BF16.gguf G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX
```

### Other ROCmFPX ftypes you may be asked for
`Q4_0_ROCMFP4`, `Q4_0_ROCMFP4_FAST` (4.25 bpw, speed-first), `Q4_0_ROCMFP4_STRIX(_LEAN)`,
`Q4_0_ROCMFP4_COHERENT`, `Q8_0_ROCMFPX`, `Q6_0_ROCMFPX`, `Q3_0_ROCMFPX`, `Q2_0_ROCMFPX` (2.5 bpw).
List all with `./build-cpu/bin/llama-quantize --help`.

---

## 8. VRAM-fit hybrid recipe (q4 bulk + q8 sensitive), e.g. ~16 GB target

When the file must fit a smaller GPU (e.g. 24 GB laptop VRAM minus KV cache ⇒ ~16 GB file),
invert the recipe: **base ftype `Q4_0_ROCMFP4_FAST`**, then override *sensitive* tensors to
`q8_0_rocmfpx`. To pick the sensitive set, mirror the tiers of Unsloth's Dynamic recipe
(e.g. `UD-Q4_K_XL`) for the same model:

1. Download the reference quant and tabulate per-tensor types:
   ```python
   from gguf import GGUFReader
   import re, collections
   r = GGUFReader("<UD-Q4_K_XL>.gguf", "r")
   g = collections.defaultdict(set)
   for t in r.tensors:
       g[re.sub(r'^blk\.\d+', 'blk.N', t.name)].add(t.tensor_type.name)
   for k in sorted(g): print(k, sorted(g[k]))
   ```
2. Map tiers: Unsloth's **Q6_K tensors = most sensitive** (typically `attn_v`, `output.weight`,
   MTP/nextn projections), **Q5_K = next** (attn q/k/qkv/gate/output, `ffn_up/down`, SSM/`ssm_out`),
   **IQ4_XS/Q4_K = bulk** (`ffn_gate`, `token_embd`).
3. Budget math: each tensor bumped q4→q8 costs `params × 0.5` bytes (8.25 vs 4.25 bpw).
   Count instances per group from the BF16 shards, then greedily add from most sensitive
   until file size ≈ target (dry-run reports exact `quant size`).

Worked example — Qwen3.8-27B (`qwen35` arch, 65 layers: 17 full-attn + 48 linear-attn/SSM,
dense FFN, MTP `nextn` head), 16 GB target:

```bash
./build-cpu/bin/llama-quantize \
  --tensor-type "attn_q.weight=q8_0_rocmfpx" \
  --tensor-type "attn_k.weight=q8_0_rocmfpx" \
  --tensor-type "attn_v.weight=q8_0_rocmfpx" \
  --tensor-type "attn_output.weight=q8_0_rocmfpx" \
  --tensor-type "attn_gate.weight=q8_0_rocmfpx" \
  --tensor-type "ssm_out.weight=q8_0_rocmfpx" \
  --tensor-type "^output.weight=q8_0_rocmfpx" \
  Qwen3.8-27B-BF16-00001-of-00002.gguf Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf Q4_0_ROCMFP4_FAST
# → 16773 MiB (5.15 bpw): 340× q4_0_rocmfp4_fast + 165× q8_0_rocmfpx + 360× f32
```

Notes for this recipe:
- Use `^output.weight` (anchored regex) — plain `output` would also match `attn_output`.
- With base `Q4_0_ROCMFP4_FAST`, the quantizer **auto-bumps MTP/draft-sensitive projections**
  (`nextn.eh_proj` etc.) to plain `q8_0` — expected and desirable.
- `attn_qkv` (fused QKV of linear-attn layers) was left at q4 here to stay near budget;
  it's the first thing to bump if you have ~1.2 GiB more headroom.
- `token_embd` stays q4 — Unsloth's own XL recipe keeps it at Q4_K.
- For MoE models, apply both sections: experts to q4 (§3/§7) *and* sensitive non-expert
  tensors to q8 — the regexes compose freely.

### Gotchas
- Install the **repo's** `gguf-py` (`pip install -e gguf-py`); the stock pip `gguf` raises
  `ValueError: ... is not a valid GGMLQuantizationType` on these types.
- `--output-tensor-type` only touches `output.weight` — the base ftype defines "everything else".
- Dense leading layers (e.g. Laguna blk.0 `ffn_gate/up/down`) are not routed experts and correctly
  stay `q8_0_rocmfpx`.
- Pass the **first** shard as input; llama.cpp picks up `-0000N-of-0000M` siblings automatically.
- Quantize from BF16/F16 sources for quality; never requantize an already-quantized GGUF.