Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>KingJones — ROCmFP4 & NVFP4 quant lab</title> | |
| <meta name="description" content="ROCmFP4 quantized models for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) and NVFP4 for NVIDIA. Measured results, including the ones that did not work."> | |
| <style> | |
| :root{ | |
| --bg:#0d1117; --panel:#151b23; --line:#2a3441; --tx:#e6edf3; --dim:#9aa7b4; | |
| --accent:#ff8a3d; --good:#3fb950; --bad:#f85149; --link:#79c0ff; | |
| } | |
| @media (prefers-color-scheme: light){ | |
| :root{ --bg:#ffffff; --panel:#f6f8fa; --line:#d8dee4; --tx:#1f2328; | |
| --dim:#59636e; --accent:#bc4c00; --good:#1a7f37; --bad:#cf222e; --link:#0969da; } | |
| } | |
| *{box-sizing:border-box} | |
| body{margin:0;background:var(--bg);color:var(--tx); | |
| font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Helvetica,Arial,sans-serif} | |
| .wrap{max-width:920px;margin:0 auto;padding:56px 22px 90px} | |
| h1{font-size:2.6rem;line-height:1.1;margin:0 0 .4rem;letter-spacing:-.02em} | |
| h2{font-size:1.35rem;margin:3rem 0 .9rem;letter-spacing:-.01em; | |
| padding-bottom:.45rem;border-bottom:1px solid var(--line)} | |
| h3{font-size:1.02rem;margin:1.8rem 0 .5rem;color:var(--dim); | |
| text-transform:uppercase;letter-spacing:.08em} | |
| p{margin:0 0 1rem} | |
| a{color:var(--link);text-decoration:none} | |
| a:hover{text-decoration:underline} | |
| .lede{font-size:1.2rem;color:var(--dim);margin-bottom:1.6rem} | |
| .rule{color:var(--accent);font-weight:600} | |
| .stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(132px,1fr)); | |
| gap:12px;margin:1.8rem 0 .5rem} | |
| .stat{background:var(--panel);border:1px solid var(--line);border-radius:10px; | |
| padding:14px 16px} | |
| .stat b{display:block;font-size:1.7rem;line-height:1.15;letter-spacing:-.02em} | |
| .stat span{font-size:.78rem;color:var(--dim);text-transform:uppercase; | |
| letter-spacing:.06em} | |
| .tablewrap{overflow-x:auto;-webkit-overflow-scrolling:touch;margin:1rem 0} | |
| table{border-collapse:collapse;width:100%;font-size:.92rem;min-width:560px} | |
| th,td{padding:9px 12px;border-bottom:1px solid var(--line);text-align:left} | |
| th{color:var(--dim);font-weight:600;font-size:.78rem;text-transform:uppercase; | |
| letter-spacing:.05em;white-space:nowrap} | |
| td.num,th.num{text-align:right;font-variant-numeric:tabular-nums} | |
| .win{color:var(--good);font-weight:600} | |
| .loss{color:var(--bad);font-weight:600} | |
| code{background:var(--panel);border:1px solid var(--line);border-radius:5px; | |
| padding:1px 6px;font-size:.86em; | |
| font-family:ui-monospace,SFMono-Regular,Menlo,monospace} | |
| .note{background:var(--panel);border-left:3px solid var(--accent); | |
| border-radius:0 8px 8px 0;padding:14px 18px;margin:1.3rem 0} | |
| .note p:last-child{margin-bottom:0} | |
| ul{padding-left:1.15rem;margin:0 0 1rem} | |
| li{margin-bottom:.55rem} | |
| .pills{display:flex;flex-wrap:wrap;gap:7px;margin:.7rem 0 1.4rem} | |
| .pill{background:var(--panel);border:1px solid var(--line);border-radius:999px; | |
| padding:4px 12px;font-size:.82rem;color:var(--dim)} | |
| footer{margin-top:3.5rem;padding-top:1.4rem;border-top:1px solid var(--line); | |
| color:var(--dim);font-size:.88rem} | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <h1>KingJones</h1> | |
| <p class="lede">Quantizing large models for hardware most people don't benchmark on — | |
| and publishing the results that <em>didn't</em> work.</p> | |
| <div class="pills"> | |
| <span class="pill">Ryzen AI Max+ 395</span> | |
| <span class="pill">Strix Halo · gfx1151</span> | |
| <span class="pill">Radeon 8060S</span> | |
| <span class="pill">ROCm 7.2.4</span> | |
| <span class="pill">128 GB unified</span> | |
| <span class="pill">NVFP4 · NVIDIA</span> | |
| </div> | |
| <div class="stats"> | |
| <div class="stat"><b>18</b><span>repositories</span></div> | |
| <div class="stat"><b>846</b><span>GiB published</span></div> | |
| <div class="stat"><b>14</b><span>base models</span></div> | |
| <div class="stat"><b>13</b><span>ROCmFP4 builds</span></div> | |
| </div> | |
| <h2>What I actually found</h2> | |
| <p>ROCmFP4 does <span class="rule">not</span> universally speed up decode. That's the | |
| most useful thing I can tell you, and it took four builds and one discarded model | |
| to learn it.</p> | |
| <div class="tablewrap"> | |
| <table> | |
| <thead><tr> | |
| <th>Model</th><th>Architecture</th><th class="num">Active</th> | |
| <th class="num">Decode, short ctx</th><th class="num">Decode, long ctx</th><th>Verdict</th> | |
| </tr></thead> | |
| <tbody> | |
| <tr><td>Laguna-S-2.1 118B-A8B</td><td><code>laguna</code></td><td class="num">~8B</td> | |
| <td class="num win">+62.6%</td><td class="num win">+43.6%</td><td>ship it</td></tr> | |
| <tr><td>Step-3.7-Flash 198B MoE</td><td><code>step35</code></td><td class="num">~11B</td> | |
| <td class="num win">+18%</td><td class="num win">+20%</td><td>ship it</td></tr> | |
| <tr><td>Leanstral-1.5 119B-A6B</td><td><code>deepseek2</code> · MLA</td><td class="num">~6.5B</td> | |
| <td class="num">+1.5%</td><td class="num loss">−8.2%</td><td>size only</td></tr> | |
| <tr><td>KAT-Coder-V2.5-Dev 35B-A3B</td><td>hybrid linear</td><td class="num">~3B</td> | |
| <td class="num">+12%</td><td class="num loss">−37%</td><td><strong>discarded</strong></td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p>The builds that gained were the ones pushing more active parameters through the | |
| FP4 FFN kernels. MLA and hybrid-linear attention shift work <em>away</em> from that | |
| path and gained nothing — one regressed badly as context grew.</p> | |
| <div class="note"> | |
| <p><strong>Benchmarking at one context length will lie to you.</strong> KAT-Coder | |
| looks like a 12% win at short context. At long context it's 37% slower. I nearly | |
| shipped it.</p> | |
| <p><strong>Attention type and active-param count are confounded in this sample.</strong> | |
| Four data points is a hypothesis, not a proof. I'd rather it be tested than believed.</p> | |
| </div> | |
| <h2>How I measure</h2> | |
| <p>Numbers are worth exactly as much as the discipline behind them.</p> | |
| <ul> | |
| <li><strong>Equal generation length</strong> — every run emits exactly 256 tokens with | |
| <code>ignore_eos</code>. Comparing tok/s across runs with different token counts is | |
| meaningless; one early run of mine was soft because a baseline stopped after 4 tokens.</li> | |
| <li><strong>Nonce-prefixed prompts</strong> — every prompt opens with a fresh UUID so the | |
| prefix cache can't serve a seen prefix and inflate prefill. <code>cached_tokens</code> | |
| is recorded each run so you can check it was 0. I caught a contaminated campaign this way.</li> | |
| <li><strong>Two context lengths minimum</strong> — see KAT-Coder.</li> | |
| <li><strong>One model resident, cold-loaded, three runs, median</strong> — on 128 GB | |
| unified memory two large models won't coexist, and a half-swapped model produces noise.</li> | |
| <li><strong>Failures published</strong> — OOM, crash and regression get a row in the | |
| table, not a deletion. Where a run shared the machine with other traffic, the card | |
| says the ordering is reliable but the exact ratios aren't.</li> | |
| </ul> | |
| <h2>The builds</h2> | |
| <h3>ROCmFP4 — AMD Strix Halo</h3> | |
| <p>Requires the <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX fork</a>. | |
| These will not load in stock llama.cpp, Ollama or LM Studio.</p> | |
| <div class="tablewrap"> | |
| <table> | |
| <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Contents</th></tr></thead> | |
| <tbody> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Muse-Glimmer-30B-ROCmFP4-Strix-Halo-DFlash-GGUF">Muse-Glimmer-30B</a></td><td>meta-models</td><td class="num">63.0 GiB</td><td>4 variants · drafter · vision</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Laguna-S-2.1-ROCmFP4-STRIX_LEAN-GGUF">Laguna-S-2.1</a></td><td>poolside</td><td class="num">58.3 GiB</td><td>STRIX_LEAN</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Leanstral-1.5-119B-A6B-ROCmFP4-STRIX_LEAN-GGUF">Leanstral-1.5-119B-A6B</a></td><td>mistralai</td><td class="num">59.0 GiB</td><td>STRIX_LEAN</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Step-3.7-Flash-ROCmFP4-STRIX_LEAN-GGUF">Step-3.7-Flash</a></td><td>stepfun-ai</td><td class="num">101.4 GiB</td><td>STRIX_LEAN · vision</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-180B-ROCmFP4-STRIX_LEAN-GGUF">DeepSeek-V4-Flash-180B</a></td><td>deepseek-ai</td><td class="num">181.7 GiB</td><td>2 variants</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-0731-ROCmFP4">DeepSeek-V4-Flash-0731</a></td><td>deepseek-ai</td><td class="num">100.4 GiB</td><td>ROCmFP4</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Qwen3-Next-80B-A3B-Instruct-ROCmFP4-STRIX-GGUF">Qwen3-Next-80B-A3B</a></td><td>Qwen</td><td class="num">39.7 GiB</td><td>STRIX</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/BTL-4-ROCmFP4-STRIX-GGUF">BTL-4</a></td><td>badtheorylabs</td><td class="num">18.2 GiB</td><td>STRIX · vision</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/North-Mini-Code-1.0-ROCmFP4-STRIX-GGUF">North-Mini-Code-1.0</a></td><td>CohereLabs</td><td class="num">15.3 GiB</td><td>STRIX</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/GLM-4.7-Flash-ROCmFP4-STRIX-GGUF">GLM-4.7-Flash</a></td><td>zai-org</td><td class="num">14.9 GiB</td><td>STRIX</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-ROCmFP4-STRIX-GGUF">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">8.0 GiB</td><td>STRIX</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Instella-MoE-16B-A3B-Think-ROCmFP4-STRIX-GGUF">Instella-MoE-16B-A3B-Think</a></td><td>amd</td><td class="num">7.9 GiB</td><td>STRIX</td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <h3>NVFP4 — NVIDIA</h3> | |
| <div class="tablewrap"> | |
| <table> | |
| <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Ling-3.0-flash-NVFP4-SGLang-MTP">Ling-3.0-flash</a></td><td>inclusionAI</td><td class="num">75.8 GiB</td><td>SGLang · MTP working</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Frontis-MA1-35B-NVFP4">Frontis-MA1-35B</a></td><td>FrontisAI</td><td class="num">23.3 GiB</td><td>with MTP</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Macaron-V1-Tall-NVFP4">Macaron-V1-Tall</a></td><td>mindlab-research</td><td class="num">23.3 GiB</td><td>MTP · LoRA-compatible</td></tr> | |
| <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-NVFP4">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">9.0 GiB</td><td>tool calling</td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <h2>Things that cost me time, so they don't cost you any</h2> | |
| <ul> | |
| <li><strong>A vision port can compile cleanly and be completely wrong.</strong> | |
| Muse Glimmer needs per-layer sparse-window attention masks. Drop them and the build | |
| works, runs, and describes images fluently and incorrectly. A four-quadrant colour | |
| test catches it — a broken mask names the colours right and puts them in the wrong | |
| corners. "Describe this image" does not catch it.</li> | |
| <li><strong>Per-token acceptance is the wrong metric for a block drafter.</strong> | |
| DFlash proposes 15 tokens per pass; ~21% acceptance sounds broken and actually means | |
| ~4.13 tokens landing per target pass, which is healthy. Read tokens-per-pass.</li> | |
| <li><strong>A higher acceptance rate can be slower.</strong> n-gram speculation hit | |
| 42.9% acceptance against DFlash's 21% and ran 45% slower, because it proposes far | |
| fewer tokens per pass.</li> | |
| <li><strong>Check what else is on the box.</strong> An identical request measured 40s | |
| on a quiet machine and 118s while another model was resident. Contention will invent | |
| a result and hand it to you with a straight face.</li> | |
| </ul> | |
| <h2>Credit</h2> | |
| <p>The ROCmFP4 / ROCmFPX tensor formats are <strong>not my work</strong> — they're | |
| <a href="https://github.com/charlie12345">charlie12345</a> / <code>caf</code>'s | |
| <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX</a> fork of llama.cpp, with | |
| contributions from <code>ciru-ai</code>, Tom Turney, <code>PlunderStruck</code> and | |
| Aydan S. Every ROCmFP4 file here was produced with their quantizer and runs on their | |
| runtime. If these builds are useful to you, star that repo.</p> | |
| <p>Built on <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> and | |
| <a href="https://github.com/ROCm/ROCm">AMD ROCm</a>. All base model weights, licences | |
| and capabilities belong to their original authors — I contribute quantisation and | |
| measurement only.</p> | |
| <footer> | |
| <p><a href="https://huggingface.co/kingjones777">All models on Hugging Face</a> · | |
| <a href="https://github.com/kingjones30/strix-halo-quant-lab">Benchmark harness & raw results</a></p> | |
| <p>If you reproduce one of these builds on your own hardware, please open a discussion | |
| on the model — independent numbers are worth more than another round of mine.</p> | |
| </footer> | |
| </div> | |
| </body> | |
| </html> | |