Hugging Face's logo

JackBinary
/
ROCmFPX-Quantize-Playbook

GGUF
rocmfpx
quantization
llama-cpp
playbook
Model card Files
xet
 Community
ROCmFPX-Quantize-Playbook / rocmfpx-quantization-guide.md
JackBinary's picture
JackBinary
Upload folder using huggingface_hub
97785d7 verified
preview code
|
Raw
History Blame Contribute Delete
11.3 kB

ROCmFPX Quantization Guide — q4_0_rocmfp4_fast experts + q8_0_rocmfpx everything else

Self-contained playbook for producing "hybrid" ROCmFPX GGUF quants from a Hugging Face repo link: routed MoE experts (up/down/gate) → q4_0_rocmfp4_fast, every other tensor → q8_0_rocmfpx. Runs fully CPU-only (no GPU needed; quantization is a CPU reference path).

Validated end-to-end on:

Model Source Recipe applied to Result
Laguna-S-2.1 (118B-A10B, poolside, arch laguna) unsloth/Laguna-S-2.1-GGUF/BF16 (5 shards) ffn_{gate,up,down}_exps 224 GB BF16 → 61.6 GB (4.39 bpw)
Qwen3.8-27B (dense) unsloth/Qwen3.8-27B-GGUF/BF16 (2 shards) n/a (dense) 52 GB BF16 → 26.9 GB (8.25 bpw)
G4-MeroMero-26B-A4B (Gemma4 MoE) raw HF weights llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic ffn_gate_up_exps, ffn_down_exps converted to BF16 GGUF (50.5 GB) → 14.0 GB (4.64 bpw)
Qwen3.8-27B hybrid (dense, qwen35 arch) unsloth/Qwen3.8-27B-GGUF/BF16 sensitive tensors per Unsloth UD-Q4_K_XL (see §8) 16.4 GB (5.15 bpw), targets ~16 GB VRAM-fit file

0. One-time setup (CPU-only) 

# System deps: gcc, cmake, git, python3 + pip
git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX

# CPU-only build (no HIP/CUDA/Vulkan/Metal)
cmake -B build-cpu -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
      -DGGML_METAL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build-cpu -j "$(nproc)" --target llama-quantize llama-gguf-split llama-gguf

# Python deps for HF→GGUF conversion (only needed for Workflow B)
pip install -r requirements/requirements-convert_hf_to_gguf.txt
pip install -e gguf-py          # install the repo's gguf lib (older pip gguf can't read ROCmFPX types!)

Binaries land in build-cpu/bin/: llama-quantize, llama-gguf-split, llama-gguf.

Note: the ROCmFPX quant types (q4_0_rocmfp4_fast = 101, q8_0_rocmfpx = 103, etc.) are only understood by this fork's tooling and recent-ish llama.cpp. Use the binaries built here.

1. Decide which workflow applies

Look at the HF repo (https://huggingface.co/<repo>):

  • Workflow A — repo already contains GGUF files, e.g. BF16/*-00001-of-0000N.gguf (typical for unsloth/*-GGUF repos). Skip straight to quantizing.
  • Workflow B — repo contains raw weights (model*.safetensors + config.json). Convert to BF16 GGUF first, then quantize.

Check sizes before downloading:

curl -s https://huggingface.co/api/models/<ORG>/<REPO>/tree/main/BF16 | \
  python3 -c "import json,sys; [print(s['path'], round(s['size']/1e9,2),'GB') for s in json.load(sys.stdin)]"

Download with hf CLI (shards download in parallel; pass the first shard to llama.cpp later — remaining shards are discovered automatically):

pip install -q huggingface_hub
hf download <ORG>/<REPO> --repo-type model --include "BF16/*" --local-dir /workspace/models/<NAME>

2. Find the routed-expert tensor names (critical step)

Tensor naming depends on architecture. Inspect the first GGUF shard (header only — works as soon as the file starts existing, and only needs a few MB):

python3 - <<'EOF'
from gguf import GGUFReader
import re, collections
r = GGUFReader("<first-shard>.gguf", "r")
print("arch:", bytes(r.fields['general.architecture'].parts[-1]).decode())
names = sorted(set(re.sub(r'^blk\.\d+', 'blk.N', t.name) for t in r.tensors))
for n in names: print(n)
print(collections.Counter(t.tensor_type.name for t in r.tensors))
EOF

Build the expert regex from the patterns you see:

What you see in the tensor list Expert regex for --tensor-type
blk.N.ffn_gate_exps.weight, ffn_up_exps, ffn_down_exps (packed, e.g. laguna) ffn_(gate|up|down)_exps
blk.N.ffn_gate_up_exps.weight, ffn_down_exps (fused gate+up, e.g. gemma4) ffn_(gate_up|down)_exps\.weight
blk.N.ffn_gate.M, ffn_up.M, ffn_down.M (per-expert tensors) ffn_(gate|up|down)\.\d+\.

Patterns are regexes matched against lowercased tensor names (regex_search, i.e. partial match). Norms, biases, router scales (*.scale, *.bias, 1-D tensors) are never quantized regardless, so matching a little too broadly is harmless — but still verify with a dry run.

3. Dry run (always do this first)

--output-tensor-type only affects output.weight; to get "everything else = q8_0_rocmfpx" use the base ftype Q8_0_ROCMFPX and override experts:

./build-cpu/bin/llama-quantize --dry-run \
  --tensor-type "<EXPERT_REGEX>=q4_0_rocmfp4_fast" \
  <input-BF16-shard-1.gguf> <output-name.gguf> Q8_0_ROCMFPX

Confirm in the output:

  • applying manual override: q8_0_rocmfp4_fast... lines appear for every expert tensor (count = layers × {gate,up,down}, e.g. 47 MoE layers × 3 = 141 for Laguna),
  • attention / shared experts / embeddings → (q8_0_rocmfpx),
  • norms/biases/routers stay f32,
  • the final quant size = ... (X.XX BPW) line looks sane (~4.4 bpw for mostly-expert MoE, ~8.25 bpw for dense).

For a dense model, drop --tensor-type entirely.

4. Quantize 

./build-cpu/bin/llama-quantize \
  --tensor-type "<EXPERT_REGEX>=q4_0_rocmfp4_fast" \
  <input-BF16-shard-1.gguf> <output-name.gguf> Q8_0_ROCMFPX

Flags worth knowing:

  • --keep-split — output keeps the input's shard layout.
  • Default (no flag) — auto-splits output at 50 GiB.
  • --tensor-type can be repeated, or use --tensor-type-file list.txt (one name=type per line).
  • Add trailing nthreads positional arg to cap thread count.

Speed reference (64-core server): 118B MoE ≈ 8 min, 27B dense ≈ 1 min, 26B MoE ≈ 3 min. Memory use is modest (mmap-based); RAM ≈ source size is plenty.

5. Verify + merge shards

Verify the result (works on merged or shard-1 file):

./build-cpu/bin/llama-quantize --dry-run <output.gguf> /tmp/dummy.gguf COPY | grep "type .*tensors"
# e.g.:  f32: 287 | q4_0_rocmfp4_fast: 141 | q8_0_rocmfpx: 386

Merge shards into one file (only pass the first shard; the rest are auto-discovered):

./build-cpu/bin/llama-gguf-split --merge \
  <output>-00001-of-0000N.gguf <output-merged.gguf>

(--split / --split-max-size do the reverse. Note: the llama-gguf <file> r example tool aborts on ROCmFPX types even for valid files — use the quantize-dry-run check above instead.)

6. Workflow B only: HF weights → BF16 GGUF

For repos with *.safetensors + config.json (check architectures in config.json is registered in convert_hf_to_gguf.py — e.g. Gemma4ForConditionalGeneration is supported by this fork):

hf download <ORG>/<REPO> --local-dir /workspace/models/<NAME>   # all files incl. tokenizer

python3 convert_hf_to_gguf.py /workspace/models/<NAME> \
  --outfile /workspace/models/<NAME>-BF16.gguf --outtype bf16
# for multimodal models, add --mmproj to also export the vision projector (separate file)

Then continue at Step 2 with the produced BF16 GGUF.

7. Cheat sheet of the runs documented here 

# Laguna-S-2.1 (laguna arch, packed experts)
./build-cpu/bin/llama-quantize \
  --tensor-type "ffn_(gate|up|down)_exps=q4_0_rocmfp4_fast" \
  --keep-split \
  Laguna-S-2.1-BF16-00001-of-00005.gguf Laguna-S-2.1-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX

# Qwen3.8-27B (dense — pure q8_0_rocmfpx)
./build-cpu/bin/llama-quantize \
  Qwen3.8-27B-BF16-00001-of-00002.gguf Qwen3.8-27B-Q8_0_ROCMFPX.gguf Q8_0_ROCMFPX

# Gemma4 MoE (fused gate_up experts), after convert_hf_to_gguf.py
./build-cpu/bin/llama-quantize \
  --tensor-type "ffn_(gate_up|down)_exps.weight=q4_0_rocmfp4_fast" \
  G4-MeroMero-26B-A4B-BF16.gguf G4-MeroMero-26B-A4B-Q8_0_ROCMFPX-Q4FAST-experts.gguf Q8_0_ROCMFPX

Other ROCmFPX ftypes you may be asked for

Q4_0_ROCMFP4, Q4_0_ROCMFP4_FAST (4.25 bpw, speed-first), Q4_0_ROCMFP4_STRIX(_LEAN), Q4_0_ROCMFP4_COHERENT, Q8_0_ROCMFPX, Q6_0_ROCMFPX, Q3_0_ROCMFPX, Q2_0_ROCMFPX (2.5 bpw). List all with ./build-cpu/bin/llama-quantize --help.

8. VRAM-fit hybrid recipe (q4 bulk + q8 sensitive), e.g. ~16 GB target

When the file must fit a smaller GPU (e.g. 24 GB laptop VRAM minus KV cache ⇒ ~16 GB file), invert the recipe: base ftype Q4_0_ROCMFP4_FAST, then override sensitive tensors to q8_0_rocmfpx. To pick the sensitive set, mirror the tiers of Unsloth's Dynamic recipe (e.g. UD-Q4_K_XL) for the same model:

  1. Download the reference quant and tabulate per-tensor types:
    from gguf import GGUFReader
import re, collections
r = GGUFReader("<UD-Q4_K_XL>.gguf", "r")
g = collections.defaultdict(set)
for t in r.tensors:
    g[re.sub(r'^blk\.\d+', 'blk.N', t.name)].add(t.tensor_type.name)
for k in sorted(g): print(k, sorted(g[k]))
  2. Map tiers: Unsloth's Q6_K tensors = most sensitive (typically attn_v, output.weight, MTP/nextn projections), Q5_K = next (attn q/k/qkv/gate/output, ffn_up/down, SSM/ssm_out), IQ4_XS/Q4_K = bulk (ffn_gate, token_embd).
  3. Budget math: each tensor bumped q4→q8 costs params × 0.5 bytes (8.25 vs 4.25 bpw). Count instances per group from the BF16 shards, then greedily add from most sensitive until file size ≈ target (dry-run reports exact quant size).

Worked example — Qwen3.8-27B (qwen35 arch, 65 layers: 17 full-attn + 48 linear-attn/SSM, dense FFN, MTP nextn head), 16 GB target:

./build-cpu/bin/llama-quantize \
  --tensor-type "attn_q.weight=q8_0_rocmfpx" \
  --tensor-type "attn_k.weight=q8_0_rocmfpx" \
  --tensor-type "attn_v.weight=q8_0_rocmfpx" \
  --tensor-type "attn_output.weight=q8_0_rocmfpx" \
  --tensor-type "attn_gate.weight=q8_0_rocmfpx" \
  --tensor-type "ssm_out.weight=q8_0_rocmfpx" \
  --tensor-type "^output.weight=q8_0_rocmfpx" \
  Qwen3.8-27B-BF16-00001-of-00002.gguf Qwen3.8-27B-Q4FAST-Q8-sensitive.gguf Q4_0_ROCMFP4_FAST
# → 16773 MiB (5.15 bpw): 340× q4_0_rocmfp4_fast + 165× q8_0_rocmfpx + 360× f32

Notes for this recipe:

  • Use ^output.weight (anchored regex) — plain output would also match attn_output.
  • With base Q4_0_ROCMFP4_FAST, the quantizer auto-bumps MTP/draft-sensitive projections (nextn.eh_proj etc.) to plain q8_0 — expected and desirable.
  • attn_qkv (fused QKV of linear-attn layers) was left at q4 here to stay near budget; it's the first thing to bump if you have ~1.2 GiB more headroom.
  • token_embd stays q4 — Unsloth's own XL recipe keeps it at Q4_K.
  • For MoE models, apply both sections: experts to q4 (§3/§7) and sensitive non-expert tensors to q8 — the regexes compose freely.

Gotchas

  • Install the repo's gguf-py (pip install -e gguf-py); the stock pip gguf raises ValueError: ... is not a valid GGMLQuantizationType on these types.
  • --output-tensor-type only touches output.weight — the base ftype defines "everything else".
  • Dense leading layers (e.g. Laguna blk.0 ffn_gate/up/down) are not routed experts and correctly stay q8_0_rocmfpx.
  • Pass the first shard as input; llama.cpp picks up -0000N-of-0000M siblings automatically.
  • Quantize from BF16/F16 sources for quality; never requantize an already-quantized GGUF.