| --- |
| tags: |
| - rocmfpx |
| - gguf |
| - quantization |
| - llama-cpp |
| - playbook |
| license: mit |
| --- |
| |
| # ROCmFPX Quantize Playbook |
|
|
| A field-tested, agent-ready playbook for producing **ROCmFPX hybrid GGUF quantizations** |
| (`q4_0_rocmfp4_fast` / `q8_0_rocmfpx`) with the |
| [ROCmFPX fork of llama.cpp](https://github.com/charlie12345/ROCmFPX) — fully CPU-only, |
| from either pre-quantized GGUF repos (e.g. Unsloth BF16) or raw HF safetensors. |
|
|
| The complete guide is in [`rocmfpx-quantization-guide.md`](./rocmfpx-quantization-guide.md). |
| Hand this repo (or just that file) to a coding agent together with a Hugging Face link and a |
| recipe, and it can reproduce every step below. |
|
|
| ## What it covers |
|
|
| - One-time CPU-only build of `llama-quantize` / `llama-gguf-split` + Python conversion deps |
| - **Workflow A:** source repo already has BF16 GGUF shards (e.g. `unsloth/*-GGUF`) |
| - **Workflow B:** raw safetensors → `convert_hf_to_gguf.py` → BF16 GGUF |
| - How to discover routed-expert tensor names per architecture (regex lookup table) |
| - Dry-run verification, shard merging, and output validation |
| - VRAM-fit hybrid recipe (q4 bulk + q8 sensitive) derived from Unsloth Dynamic tiers |
| - Cheatsheets, timings, and gotchas (regex anchoring, MTP auto-protection, gguf-py version trap…) |
|
|
| ## Validated runs |
|
|
| | Model | Recipe | Result | |
| |---|---|---| |
| | [Laguna-S-2.1 118B-A10B](https://huggingface.co/JackBinary/Laguna-S-2.1-GGUF-ROCMFPX) | q4 experts / q8 rest | 224 GB → 61.6 GB (4.39 bpw) | |
| | [Qwen3.8-27B](https://huggingface.co/JackBinary/Qwen3.8-27B-GGUF-ROCMFPX) | pure q8 **and** 16 GB hybrid from UD-Q4_K_XL tiers | 26.9 GB (8.25 bpw) / 16.4 GB (5.15 bpw) | |
| | [G4-MeroMero-26B-A4B](https://huggingface.co/JackBinary/G4-MeroMero-26B-A4B-GGUF-ROCMFPX) | q4 experts / q8 rest + mmproj | 50.5 GB → 14.0 GB (4.64 bpw) | |
|
|
| Timings on a 64-core CPU box: 118B MoE ≈ 8 min, 27B dense ≈ 1 min, 26B MoE ≈ 3 min. |
|
|
| ## Related |
|
|
| - Quant format project: https://github.com/charlie12345/ROCmFPX |
| - Companion model collection: [ROCmFPX GGUF Quants](https://huggingface.co/collections/JackBinary/rocmfpx-gguf-quants-6a7f51b4114f2897ea046100) |
|
|
