Spaces:

kingjones777
/

quant-lab

Running

File size: 13,369 Bytes

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>KingJones — ROCmFP4 &amp; NVFP4 quant lab</title>
<meta name="description" content="ROCmFP4 quantized models for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) and NVFP4 for NVIDIA. Measured results, including the ones that did not work.">
<style>
  :root{
    --bg:#0d1117; --panel:#151b23; --line:#2a3441; --tx:#e6edf3; --dim:#9aa7b4;
    --accent:#ff8a3d; --good:#3fb950; --bad:#f85149; --link:#79c0ff;
  }
  @media (prefers-color-scheme: light){
    :root{ --bg:#ffffff; --panel:#f6f8fa; --line:#d8dee4; --tx:#1f2328;
           --dim:#59636e; --accent:#bc4c00; --good:#1a7f37; --bad:#cf222e; --link:#0969da; }
  }
  *{box-sizing:border-box}
  body{margin:0;background:var(--bg);color:var(--tx);
       font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Helvetica,Arial,sans-serif}
  .wrap{max-width:920px;margin:0 auto;padding:56px 22px 90px}
  h1{font-size:2.6rem;line-height:1.1;margin:0 0 .4rem;letter-spacing:-.02em}
  h2{font-size:1.35rem;margin:3rem 0 .9rem;letter-spacing:-.01em;
     padding-bottom:.45rem;border-bottom:1px solid var(--line)}
  h3{font-size:1.02rem;margin:1.8rem 0 .5rem;color:var(--dim);
     text-transform:uppercase;letter-spacing:.08em}
  p{margin:0 0 1rem}
  a{color:var(--link);text-decoration:none}
  a:hover{text-decoration:underline}
  .lede{font-size:1.2rem;color:var(--dim);margin-bottom:1.6rem}
  .rule{color:var(--accent);font-weight:600}
  .stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(132px,1fr));
         gap:12px;margin:1.8rem 0 .5rem}
  .stat{background:var(--panel);border:1px solid var(--line);border-radius:10px;
        padding:14px 16px}
  .stat b{display:block;font-size:1.7rem;line-height:1.15;letter-spacing:-.02em}
  .stat span{font-size:.78rem;color:var(--dim);text-transform:uppercase;
             letter-spacing:.06em}
  .tablewrap{overflow-x:auto;-webkit-overflow-scrolling:touch;margin:1rem 0}
  table{border-collapse:collapse;width:100%;font-size:.92rem;min-width:560px}
  th,td{padding:9px 12px;border-bottom:1px solid var(--line);text-align:left}
  th{color:var(--dim);font-weight:600;font-size:.78rem;text-transform:uppercase;
     letter-spacing:.05em;white-space:nowrap}
  td.num,th.num{text-align:right;font-variant-numeric:tabular-nums}
  .win{color:var(--good);font-weight:600}
  .loss{color:var(--bad);font-weight:600}
  code{background:var(--panel);border:1px solid var(--line);border-radius:5px;
       padding:1px 6px;font-size:.86em;
       font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
  .note{background:var(--panel);border-left:3px solid var(--accent);
        border-radius:0 8px 8px 0;padding:14px 18px;margin:1.3rem 0}
  .note p:last-child{margin-bottom:0}
  ul{padding-left:1.15rem;margin:0 0 1rem}
  li{margin-bottom:.55rem}
  .pills{display:flex;flex-wrap:wrap;gap:7px;margin:.7rem 0 1.4rem}
  .pill{background:var(--panel);border:1px solid var(--line);border-radius:999px;
        padding:4px 12px;font-size:.82rem;color:var(--dim)}
  footer{margin-top:3.5rem;padding-top:1.4rem;border-top:1px solid var(--line);
         color:var(--dim);font-size:.88rem}
</style>
</head>
<body>
<div class="wrap">

  <h1>KingJones</h1>
  <p class="lede">Quantizing large models for hardware most people don't benchmark on —
  and publishing the results that <em>didn't</em> work.</p>

  <div class="pills">
    <span class="pill">Ryzen AI Max+ 395</span>
    <span class="pill">Strix Halo · gfx1151</span>
    <span class="pill">Radeon 8060S</span>
    <span class="pill">ROCm 7.2.4</span>
    <span class="pill">128 GB unified</span>
    <span class="pill">NVFP4 · NVIDIA</span>
  </div>

  <div class="stats">
    <div class="stat"><b>18</b><span>repositories</span></div>
    <div class="stat"><b>846</b><span>GiB published</span></div>
    <div class="stat"><b>14</b><span>base models</span></div>
    <div class="stat"><b>13</b><span>ROCmFP4 builds</span></div>
  </div>

  <h2>What I actually found</h2>
  <p>ROCmFP4 does <span class="rule">not</span> universally speed up decode. That's the
  most useful thing I can tell you, and it took four builds and one discarded model
  to learn it.</p>

  <div class="tablewrap">
  <table>
    <thead><tr>
      <th>Model</th><th>Architecture</th><th class="num">Active</th>
      <th class="num">Decode, short ctx</th><th class="num">Decode, long ctx</th><th>Verdict</th>
    </tr></thead>
    <tbody>
      <tr><td>Laguna-S-2.1 118B-A8B</td><td><code>laguna</code></td><td class="num">~8B</td>
          <td class="num win">+62.6%</td><td class="num win">+43.6%</td><td>ship it</td></tr>
      <tr><td>Step-3.7-Flash 198B MoE</td><td><code>step35</code></td><td class="num">~11B</td>
          <td class="num win">+18%</td><td class="num win">+20%</td><td>ship it</td></tr>
      <tr><td>Leanstral-1.5 119B-A6B</td><td><code>deepseek2</code> · MLA</td><td class="num">~6.5B</td>
          <td class="num">+1.5%</td><td class="num loss">−8.2%</td><td>size only</td></tr>
      <tr><td>KAT-Coder-V2.5-Dev 35B-A3B</td><td>hybrid linear</td><td class="num">~3B</td>
          <td class="num">+12%</td><td class="num loss">−37%</td><td><strong>discarded</strong></td></tr>
    </tbody>
  </table>
  </div>

  <p>The builds that gained were the ones pushing more active parameters through the
  FP4 FFN kernels. MLA and hybrid-linear attention shift work <em>away</em> from that
  path and gained nothing — one regressed badly as context grew.</p>

  <div class="note">
    <p><strong>Benchmarking at one context length will lie to you.</strong> KAT-Coder
    looks like a 12% win at short context. At long context it's 37% slower. I nearly
    shipped it.</p>
    <p><strong>Attention type and active-param count are confounded in this sample.</strong>
    Four data points is a hypothesis, not a proof. I'd rather it be tested than believed.</p>
  </div>

  <h2>How I measure</h2>
  <p>Numbers are worth exactly as much as the discipline behind them.</p>
  <ul>
    <li><strong>Equal generation length</strong> — every run emits exactly 256 tokens with
      <code>ignore_eos</code>. Comparing tok/s across runs with different token counts is
      meaningless; one early run of mine was soft because a baseline stopped after 4 tokens.</li>
    <li><strong>Nonce-prefixed prompts</strong> — every prompt opens with a fresh UUID so the
      prefix cache can't serve a seen prefix and inflate prefill. <code>cached_tokens</code>
      is recorded each run so you can check it was 0. I caught a contaminated campaign this way.</li>
    <li><strong>Two context lengths minimum</strong> — see KAT-Coder.</li>
    <li><strong>One model resident, cold-loaded, three runs, median</strong> — on 128 GB
      unified memory two large models won't coexist, and a half-swapped model produces noise.</li>
    <li><strong>Failures published</strong> — OOM, crash and regression get a row in the
      table, not a deletion. Where a run shared the machine with other traffic, the card
      says the ordering is reliable but the exact ratios aren't.</li>
  </ul>

  <h2>The builds</h2>

  <h3>ROCmFP4 — AMD Strix Halo</h3>
  <p>Requires the <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX fork</a>.
  These will not load in stock llama.cpp, Ollama or LM Studio.</p>
  <div class="tablewrap">
  <table>
    <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Contents</th></tr></thead>
    <tbody>
      <tr><td><a href="https://huggingface.co/kingjones777/Muse-Glimmer-30B-ROCmFP4-Strix-Halo-DFlash-GGUF">Muse-Glimmer-30B</a></td><td>meta-models</td><td class="num">63.0 GiB</td><td>4 variants · drafter · vision</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Laguna-S-2.1-ROCmFP4-STRIX_LEAN-GGUF">Laguna-S-2.1</a></td><td>poolside</td><td class="num">58.3 GiB</td><td>STRIX_LEAN</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Leanstral-1.5-119B-A6B-ROCmFP4-STRIX_LEAN-GGUF">Leanstral-1.5-119B-A6B</a></td><td>mistralai</td><td class="num">59.0 GiB</td><td>STRIX_LEAN</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Step-3.7-Flash-ROCmFP4-STRIX_LEAN-GGUF">Step-3.7-Flash</a></td><td>stepfun-ai</td><td class="num">101.4 GiB</td><td>STRIX_LEAN · vision</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-180B-ROCmFP4-STRIX_LEAN-GGUF">DeepSeek-V4-Flash-180B</a></td><td>deepseek-ai</td><td class="num">181.7 GiB</td><td>2 variants</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-0731-ROCmFP4">DeepSeek-V4-Flash-0731</a></td><td>deepseek-ai</td><td class="num">100.4 GiB</td><td>ROCmFP4</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Qwen3-Next-80B-A3B-Instruct-ROCmFP4-STRIX-GGUF">Qwen3-Next-80B-A3B</a></td><td>Qwen</td><td class="num">39.7 GiB</td><td>STRIX</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/BTL-4-ROCmFP4-STRIX-GGUF">BTL-4</a></td><td>badtheorylabs</td><td class="num">18.2 GiB</td><td>STRIX · vision</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/North-Mini-Code-1.0-ROCmFP4-STRIX-GGUF">North-Mini-Code-1.0</a></td><td>CohereLabs</td><td class="num">15.3 GiB</td><td>STRIX</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/GLM-4.7-Flash-ROCmFP4-STRIX-GGUF">GLM-4.7-Flash</a></td><td>zai-org</td><td class="num">14.9 GiB</td><td>STRIX</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-ROCmFP4-STRIX-GGUF">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">8.0 GiB</td><td>STRIX</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Instella-MoE-16B-A3B-Think-ROCmFP4-STRIX-GGUF">Instella-MoE-16B-A3B-Think</a></td><td>amd</td><td class="num">7.9 GiB</td><td>STRIX</td></tr>
    </tbody>
  </table>
  </div>

  <h3>NVFP4 — NVIDIA</h3>
  <div class="tablewrap">
  <table>
    <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Notes</th></tr></thead>
    <tbody>
      <tr><td><a href="https://huggingface.co/kingjones777/Ling-3.0-flash-NVFP4-SGLang-MTP">Ling-3.0-flash</a></td><td>inclusionAI</td><td class="num">75.8 GiB</td><td>SGLang · MTP working</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Frontis-MA1-35B-NVFP4">Frontis-MA1-35B</a></td><td>FrontisAI</td><td class="num">23.3 GiB</td><td>with MTP</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Macaron-V1-Tall-NVFP4">Macaron-V1-Tall</a></td><td>mindlab-research</td><td class="num">23.3 GiB</td><td>MTP · LoRA-compatible</td></tr>
      <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-NVFP4">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">9.0 GiB</td><td>tool calling</td></tr>
    </tbody>
  </table>
  </div>

  <h2>Things that cost me time, so they don't cost you any</h2>
  <ul>
    <li><strong>A vision port can compile cleanly and be completely wrong.</strong>
      Muse Glimmer needs per-layer sparse-window attention masks. Drop them and the build
      works, runs, and describes images fluently and incorrectly. A four-quadrant colour
      test catches it — a broken mask names the colours right and puts them in the wrong
      corners. "Describe this image" does not catch it.</li>
    <li><strong>Per-token acceptance is the wrong metric for a block drafter.</strong>
      DFlash proposes 15 tokens per pass; ~21% acceptance sounds broken and actually means
      ~4.13 tokens landing per target pass, which is healthy. Read tokens-per-pass.</li>
    <li><strong>A higher acceptance rate can be slower.</strong> n-gram speculation hit
      42.9% acceptance against DFlash's 21% and ran 45% slower, because it proposes far
      fewer tokens per pass.</li>
    <li><strong>Check what else is on the box.</strong> An identical request measured 40s
      on a quiet machine and 118s while another model was resident. Contention will invent
      a result and hand it to you with a straight face.</li>
  </ul>

  <h2>Credit</h2>
  <p>The ROCmFP4 / ROCmFPX tensor formats are <strong>not my work</strong> — they're
  <a href="https://github.com/charlie12345">charlie12345</a> / <code>caf</code>'s
  <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX</a> fork of llama.cpp, with
  contributions from <code>ciru-ai</code>, Tom Turney, <code>PlunderStruck</code> and
  Aydan S. Every ROCmFP4 file here was produced with their quantizer and runs on their
  runtime. If these builds are useful to you, star that repo.</p>
  <p>Built on <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> and
  <a href="https://github.com/ROCm/ROCm">AMD ROCm</a>. All base model weights, licences
  and capabilities belong to their original authors — I contribute quantisation and
  measurement only.</p>

  <footer>
    <p><a href="https://huggingface.co/kingjones777">All models on Hugging Face</a> ·
    <a href="https://github.com/kingjones30/strix-halo-quant-lab">Benchmark harness &amp; raw results</a></p>
    <p>If you reproduce one of these builds on your own hardware, please open a discussion
    on the model — independent numbers are worth more than another round of mine.</p>
  </footer>

</div>
</body>
</html>