Spaces:

kingjones777
/

quant-lab

Running

App Files Files Community

kingjones777 commited on 5 days ago

Commit

d27bcbc

verified ·

1 Parent(s): b2fe02d

space: homepage

Browse files

Files changed (1) hide show

index.html +213 -18

index.html CHANGED Viewed

@@ -1,19 +1,214 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
 </html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>KingJones — ROCmFP4 &amp; NVFP4 quant lab</title>
+<meta name="description" content="ROCmFP4 quantized models for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) and NVFP4 for NVIDIA. Measured results, including the ones that did not work.">
+<style>
+  :root{
+    --bg:#0d1117; --panel:#151b23; --line:#2a3441; --tx:#e6edf3; --dim:#9aa7b4;
+    --accent:#ff8a3d; --good:#3fb950; --bad:#f85149; --link:#79c0ff;
+  }
+  @media (prefers-color-scheme: light){
+    :root{ --bg:#ffffff; --panel:#f6f8fa; --line:#d8dee4; --tx:#1f2328;
+           --dim:#59636e; --accent:#bc4c00; --good:#1a7f37; --bad:#cf222e; --link:#0969da; }
+  }
+  *{box-sizing:border-box}
+  body{margin:0;background:var(--bg);color:var(--tx);
+       font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Helvetica,Arial,sans-serif}
+  .wrap{max-width:920px;margin:0 auto;padding:56px 22px 90px}
+  h1{font-size:2.6rem;line-height:1.1;margin:0 0 .4rem;letter-spacing:-.02em}
+  h2{font-size:1.35rem;margin:3rem 0 .9rem;letter-spacing:-.01em;
+     padding-bottom:.45rem;border-bottom:1px solid var(--line)}
+  h3{font-size:1.02rem;margin:1.8rem 0 .5rem;color:var(--dim);
+     text-transform:uppercase;letter-spacing:.08em}
+  p{margin:0 0 1rem}
+  a{color:var(--link);text-decoration:none}
+  a:hover{text-decoration:underline}
+  .lede{font-size:1.2rem;color:var(--dim);margin-bottom:1.6rem}
+  .rule{color:var(--accent);font-weight:600}
+  .stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(132px,1fr));
+         gap:12px;margin:1.8rem 0 .5rem}
+  .stat{background:var(--panel);border:1px solid var(--line);border-radius:10px;
+        padding:14px 16px}
+  .stat b{display:block;font-size:1.7rem;line-height:1.15;letter-spacing:-.02em}
+  .stat span{font-size:.78rem;color:var(--dim);text-transform:uppercase;
+             letter-spacing:.06em}
+  .tablewrap{overflow-x:auto;-webkit-overflow-scrolling:touch;margin:1rem 0}
+  table{border-collapse:collapse;width:100%;font-size:.92rem;min-width:560px}
+  th,td{padding:9px 12px;border-bottom:1px solid var(--line);text-align:left}
+  th{color:var(--dim);font-weight:600;font-size:.78rem;text-transform:uppercase;
+     letter-spacing:.05em;white-space:nowrap}
+  td.num,th.num{text-align:right;font-variant-numeric:tabular-nums}
+  .win{color:var(--good);font-weight:600}
+  .loss{color:var(--bad);font-weight:600}
+  code{background:var(--panel);border:1px solid var(--line);border-radius:5px;
+       padding:1px 6px;font-size:.86em;
+       font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
+  .note{background:var(--panel);border-left:3px solid var(--accent);
+        border-radius:0 8px 8px 0;padding:14px 18px;margin:1.3rem 0}
+  .note p:last-child{margin-bottom:0}
+  ul{padding-left:1.15rem;margin:0 0 1rem}
+  li{margin-bottom:.55rem}
+  .pills{display:flex;flex-wrap:wrap;gap:7px;margin:.7rem 0 1.4rem}
+  .pill{background:var(--panel);border:1px solid var(--line);border-radius:999px;
+        padding:4px 12px;font-size:.82rem;color:var(--dim)}
+  footer{margin-top:3.5rem;padding-top:1.4rem;border-top:1px solid var(--line);
+         color:var(--dim);font-size:.88rem}
+</style>
+</head>
+<body>
+<div class="wrap">
+  <h1>KingJones</h1>
+  <p class="lede">Quantizing large models for hardware most people don't benchmark on —
+  and publishing the results that <em>didn't</em> work.</p>
+  <div class="pills">
+    <span class="pill">Ryzen AI Max+ 395</span>
+    <span class="pill">Strix Halo · gfx1151</span>
+    <span class="pill">Radeon 8060S</span>
+    <span class="pill">ROCm 7.2.4</span>
+    <span class="pill">128 GB unified</span>
+    <span class="pill">NVFP4 · NVIDIA</span>
+  </div>
+  <div class="stats">
+    <div class="stat"><b>18</b><span>repositories</span></div>
+    <div class="stat"><b>846</b><span>GiB published</span></div>
+    <div class="stat"><b>14</b><span>base models</span></div>
+    <div class="stat"><b>13</b><span>ROCmFP4 builds</span></div>
+  </div>
+  <h2>What I actually found</h2>
+  <p>ROCmFP4 does <span class="rule">not</span> universally speed up decode. That's the
+  most useful thing I can tell you, and it took four builds and one discarded model
+  to learn it.</p>
+  <div class="tablewrap">
+  <table>
+    <thead><tr>
+      <th>Model</th><th>Architecture</th><th class="num">Active</th>
+      <th class="num">Decode, short ctx</th><th class="num">Decode, long ctx</th><th>Verdict</th>
+    </tr></thead>
+    <tbody>
+      <tr><td>Laguna-S-2.1 118B-A8B</td><td><code>laguna</code></td><td class="num">~8B</td>
+          <td class="num win">+62.6%</td><td class="num win">+43.6%</td><td>ship it</td></tr>
+      <tr><td>Step-3.7-Flash 198B MoE</td><td><code>step35</code></td><td class="num">~11B</td>
+          <td class="num win">+18%</td><td class="num win">+20%</td><td>ship it</td></tr>
+      <tr><td>Leanstral-1.5 119B-A6B</td><td><code>deepseek2</code> · MLA</td><td class="num">~6.5B</td>
+          <td class="num">+1.5%</td><td class="num loss">−8.2%</td><td>size only</td></tr>
+      <tr><td>KAT-Coder-V2.5-Dev 35B-A3B</td><td>hybrid linear</td><td class="num">~3B</td>
+          <td class="num">+12%</td><td class="num loss">−37%</td><td><strong>discarded</strong></td></tr>
+    </tbody>
+  </table>
+  </div>
+  <p>The builds that gained were the ones pushing more active parameters through the
+  FP4 FFN kernels. MLA and hybrid-linear attention shift work <em>away</em> from that
+  path and gained nothing — one regressed badly as context grew.</p>
+  <div class="note">
+    <p><strong>Benchmarking at one context length will lie to you.</strong> KAT-Coder
+    looks like a 12% win at short context. At long context it's 37% slower. I nearly
+    shipped it.</p>
+    <p><strong>Attention type and active-param count are confounded in this sample.</strong>
+    Four data points is a hypothesis, not a proof. I'd rather it be tested than believed.</p>
+  </div>
+  <h2>How I measure</h2>
+  <p>Numbers are worth exactly as much as the discipline behind them.</p>
+  <ul>
+    <li><strong>Equal generation length</strong> — every run emits exactly 256 tokens with
+      <code>ignore_eos</code>. Comparing tok/s across runs with different token counts is
+      meaningless; one early run of mine was soft because a baseline stopped after 4 tokens.</li>
+    <li><strong>Nonce-prefixed prompts</strong> — every prompt opens with a fresh UUID so the
+      prefix cache can't serve a seen prefix and inflate prefill. <code>cached_tokens</code>
+      is recorded each run so you can check it was 0. I caught a contaminated campaign this way.</li>
+    <li><strong>Two context lengths minimum</strong> — see KAT-Coder.</li>
+    <li><strong>One model resident, cold-loaded, three runs, median</strong> — on 128 GB
+      unified memory two large models won't coexist, and a half-swapped model produces noise.</li>
+    <li><strong>Failures published</strong> — OOM, crash and regression get a row in the
+      table, not a deletion. Where a run shared the machine with other traffic, the card
+      says the ordering is reliable but the exact ratios aren't.</li>
+  </ul>
+  <h2>The builds</h2>
+  <h3>ROCmFP4 — AMD Strix Halo</h3>
+  <p>Requires the <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX fork</a>.
+  These will not load in stock llama.cpp, Ollama or LM Studio.</p>
+  <div class="tablewrap">
+  <table>
+    <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Contents</th></tr></thead>
+    <tbody>
+      <tr><td><a href="https://huggingface.co/kingjones777/Muse-Glimmer-30B-ROCmFP4-Strix-Halo-DFlash-GGUF">Muse-Glimmer-30B</a></td><td>meta-models</td><td class="num">63.0 GiB</td><td>4 variants · drafter · vision</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Laguna-S-2.1-ROCmFP4-STRIX_LEAN-GGUF">Laguna-S-2.1</a></td><td>poolside</td><td class="num">58.3 GiB</td><td>STRIX_LEAN</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Leanstral-1.5-119B-A6B-ROCmFP4-STRIX_LEAN-GGUF">Leanstral-1.5-119B-A6B</a></td><td>mistralai</td><td class="num">59.0 GiB</td><td>STRIX_LEAN</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Step-3.7-Flash-ROCmFP4-STRIX_LEAN-GGUF">Step-3.7-Flash</a></td><td>stepfun-ai</td><td class="num">101.4 GiB</td><td>STRIX_LEAN · vision</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-180B-ROCmFP4-STRIX_LEAN-GGUF">DeepSeek-V4-Flash-180B</a></td><td>deepseek-ai</td><td class="num">181.7 GiB</td><td>2 variants</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/DeepSeek-V4-Flash-0731-ROCmFP4">DeepSeek-V4-Flash-0731</a></td><td>deepseek-ai</td><td class="num">100.4 GiB</td><td>ROCmFP4</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Qwen3-Next-80B-A3B-Instruct-ROCmFP4-STRIX-GGUF">Qwen3-Next-80B-A3B</a></td><td>Qwen</td><td class="num">39.7 GiB</td><td>STRIX</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/BTL-4-ROCmFP4-STRIX-GGUF">BTL-4</a></td><td>badtheorylabs</td><td class="num">18.2 GiB</td><td>STRIX · vision</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/North-Mini-Code-1.0-ROCmFP4-STRIX-GGUF">North-Mini-Code-1.0</a></td><td>CohereLabs</td><td class="num">15.3 GiB</td><td>STRIX</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/GLM-4.7-Flash-ROCmFP4-STRIX-GGUF">GLM-4.7-Flash</a></td><td>zai-org</td><td class="num">14.9 GiB</td><td>STRIX</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-ROCmFP4-STRIX-GGUF">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">8.0 GiB</td><td>STRIX</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Instella-MoE-16B-A3B-Think-ROCmFP4-STRIX-GGUF">Instella-MoE-16B-A3B-Think</a></td><td>amd</td><td class="num">7.9 GiB</td><td>STRIX</td></tr>
+    </tbody>
+  </table>
+  </div>
+  <h3>NVFP4 — NVIDIA</h3>
+  <div class="tablewrap">
+  <table>
+    <thead><tr><th>Model</th><th>Base</th><th class="num">Repo size</th><th>Notes</th></tr></thead>
+    <tbody>
+      <tr><td><a href="https://huggingface.co/kingjones777/Ling-3.0-flash-NVFP4-SGLang-MTP">Ling-3.0-flash</a></td><td>inclusionAI</td><td class="num">75.8 GiB</td><td>SGLang · MTP working</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Frontis-MA1-35B-NVFP4">Frontis-MA1-35B</a></td><td>FrontisAI</td><td class="num">23.3 GiB</td><td>with MTP</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Macaron-V1-Tall-NVFP4">Macaron-V1-Tall</a></td><td>mindlab-research</td><td class="num">23.3 GiB</td><td>MTP · LoRA-compatible</td></tr>
+      <tr><td><a href="https://huggingface.co/kingjones777/Instella-ToolCall-16B-A3B-NVFP4">Instella-ToolCall-16B-A3B</a></td><td>amd</td><td class="num">9.0 GiB</td><td>tool calling</td></tr>
+    </tbody>
+  </table>
+  </div>
+  <h2>Things that cost me time, so they don't cost you any</h2>
+  <ul>
+    <li><strong>A vision port can compile cleanly and be completely wrong.</strong>
+      Muse Glimmer needs per-layer sparse-window attention masks. Drop them and the build
+      works, runs, and describes images fluently and incorrectly. A four-quadrant colour
+      test catches it — a broken mask names the colours right and puts them in the wrong
+      corners. "Describe this image" does not catch it.</li>
+    <li><strong>Per-token acceptance is the wrong metric for a block drafter.</strong>
+      DFlash proposes 15 tokens per pass; ~21% acceptance sounds broken and actually means
+      ~4.13 tokens landing per target pass, which is healthy. Read tokens-per-pass.</li>
+    <li><strong>A higher acceptance rate can be slower.</strong> n-gram speculation hit
+      42.9% acceptance against DFlash's 21% and ran 45% slower, because it proposes far
+      fewer tokens per pass.</li>
+    <li><strong>Check what else is on the box.</strong> An identical request measured 40s
+      on a quiet machine and 118s while another model was resident. Contention will invent
+      a result and hand it to you with a straight face.</li>
+  </ul>
+  <h2>Credit</h2>
+  <p>The ROCmFP4 / ROCmFPX tensor formats are <strong>not my work</strong> — they're
+  <a href="https://github.com/charlie12345">charlie12345</a> / <code>caf</code>'s
+  <a href="https://github.com/charlie12345/ROCmFPX">ROCmFPX</a> fork of llama.cpp, with
+  contributions from <code>ciru-ai</code>, Tom Turney, <code>PlunderStruck</code> and
+  Aydan S. Every ROCmFP4 file here was produced with their quantizer and runs on their
+  runtime. If these builds are useful to you, star that repo.</p>
+  <p>Built on <a href="https://github.com/ggml-org/llama.cpp">llama.cpp</a> and
+  <a href="https://github.com/ROCm/ROCm">AMD ROCm</a>. All base model weights, licences
+  and capabilities belong to their original authors — I contribute quantisation and
+  measurement only.</p>
+  <footer>
+    <p><a href="https://huggingface.co/kingjones777">All models on Hugging Face</a> ·
+    <a href="https://github.com/kingjones30/strix-halo-quant-lab">Benchmark harness &amp; raw results</a></p>
+    <p>If you reproduce one of these builds on your own hardware, please open a discussion
+    on the model — independent numbers are worth more than another round of mine.</p>
+  </footer>
+</div>
+</body>
 </html>