Spaces:

abgunaydin
/

zero-tvm

Running

File size: 69,015 Bytes

e26c434

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Docs — Zero-TVM</title>
  <meta name="description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser on 10 hand-written WGSL kernel roles.">
  <meta name="theme-color" content="#0a0a0c">
  <link rel="icon" href="/favicon.svg" type="image/svg+xml">
  <meta property="og:type" content="article">
  <meta property="og:url" content="https://zerotvm.com/docs.html">
  <meta property="og:title" content="Docs — Zero-TVM">
  <meta property="og:description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser on 10 hand-written WGSL kernel roles.">
  <meta property="og:image" content="https://zerotvm.com/og.png">
  <meta name="twitter:card" content="summary_large_image">
  <meta name="twitter:title" content="Docs — Zero-TVM">
  <meta name="twitter:description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser on 10 hand-written WGSL kernel roles.">
  <meta name="twitter:image" content="https://zerotvm.com/og.png">
  <style>
    *, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; }

    :root {
      --bg:       #0a0a0c;
      --surface:  #111114;
      --surface2: #18181c;
      --border:   #1e1e24;
      --accent:   #6c63ff;
      --green:    #00d4aa;
      --yellow:   #f5c542;
      --red:      #ff5f57;
      --text:     #e8e8ec;
      --muted:    #666672;
      --dim:      #3a3a44;
      --sidebar-w: 260px;
      --topbar-h:  52px;
      --code-bg:  #0d0d10;
    }

    html { scroll-behavior: smooth; }

    body {
      font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
      background: var(--bg);
      color: var(--text);
      line-height: 1.65;
      display: flex;
      flex-direction: column;
      min-height: 100vh;
    }

    /* ── TOPBAR ── */
    .topbar {
      height: var(--topbar-h);
      background: var(--surface);
      border-bottom: 1px solid var(--border);
      display: flex;
      align-items: center;
      padding: 0 1.5rem;
      gap: 1.5rem;
      position: fixed;
      top: 0; left: 0; right: 0;
      z-index: 100;
    }

    .logo {
      font-weight: 800;
      font-size: 0.95rem;
      letter-spacing: -0.03em;
      color: #fff;
      text-decoration: none;
    }
    .logo span { color: var(--accent); }

    .topbar-sep { color: var(--border); }
    .topbar-title { font-size: 0.82rem; color: var(--muted); }

    .topbar-nav {
      margin-left: auto;
      display: flex;
      gap: 1.25rem;
      align-items: center;
    }

    .topbar-nav a {
      font-size: 0.8rem;
      color: var(--muted);
      text-decoration: none;
      transition: color 0.15s;
    }
    .topbar-nav a:hover { color: var(--text); }

    .topbar-cta {
      background: var(--accent);
      color: #fff !important;
      padding: 0.3rem 0.85rem;
      border-radius: 7px;
      font-weight: 600 !important;
    }

    /* ── LAYOUT ── */
    .layout {
      display: flex;
      flex: 1;
      padding-top: var(--topbar-h);
    }

    /* ── SIDEBAR ── */
    aside {
      width: var(--sidebar-w);
      min-width: var(--sidebar-w);
      background: var(--surface);
      border-right: 1px solid var(--border);
      position: fixed;
      top: var(--topbar-h);
      bottom: 0;
      left: 0;
      overflow-y: auto;
      padding: 1.5rem 0;
    }

    aside::-webkit-scrollbar { width: 4px; }
    aside::-webkit-scrollbar-thumb { background: var(--border); border-radius: 4px; }

    .sidebar-section {
      padding: 0 1rem 1rem;
    }

    .sidebar-section-label {
      font-size: 0.6rem;
      font-weight: 700;
      letter-spacing: 0.1em;
      text-transform: uppercase;
      color: var(--muted);
      padding: 0.5rem 0.5rem 0.4rem;
      display: block;
    }

    .sidebar-link {
      display: block;
      padding: 0.35rem 0.5rem;
      font-size: 0.78rem;
      color: var(--muted);
      text-decoration: none;
      border-radius: 6px;
      transition: background 0.1s, color 0.1s;
      border-left: 2px solid transparent;
    }

    .sidebar-link:hover { color: var(--text); background: var(--surface2); }
    .sidebar-link.active { color: var(--accent); border-left-color: var(--accent); background: #6c63ff0d; }
    .sidebar-link.sub { padding-left: 1.25rem; font-size: 0.73rem; }

    /* ── MAIN CONTENT ── */
    main {
      margin-left: var(--sidebar-w);
      flex: 1;
      padding: 3.5rem 3rem 6rem;
      max-width: 900px;
    }

    /* ── TYPOGRAPHY ── */
    h1 {
      font-size: 2rem;
      font-weight: 800;
      letter-spacing: -0.03em;
      margin-bottom: 0.5rem;
      line-height: 1.2;
    }

    h2 {
      font-size: 1.35rem;
      font-weight: 700;
      letter-spacing: -0.02em;
      margin-top: 3rem;
      margin-bottom: 0.75rem;
      padding-top: 2rem;
      border-top: 1px solid var(--border);
      scroll-margin-top: calc(var(--topbar-h) + 1rem);
    }

    h2:first-of-type { border-top: none; margin-top: 1.5rem; padding-top: 0; }

    h3 {
      font-size: 1rem;
      font-weight: 700;
      margin-top: 1.75rem;
      margin-bottom: 0.5rem;
      color: var(--text);
      scroll-margin-top: calc(var(--topbar-h) + 1rem);
    }

    h4 {
      font-size: 0.88rem;
      font-weight: 700;
      margin-top: 1.25rem;
      margin-bottom: 0.35rem;
      color: var(--muted);
      text-transform: uppercase;
      letter-spacing: 0.05em;
    }

    p {
      font-size: 0.9rem;
      color: #b0b0bc;
      margin-bottom: 1rem;
      line-height: 1.75;
    }

    a { color: var(--accent); text-decoration: none; }
    a:hover { text-decoration: underline; }

    ul, ol {
      margin: 0.5rem 0 1rem 1.5rem;
      font-size: 0.88rem;
      color: #b0b0bc;
      line-height: 1.8;
    }

    li { margin-bottom: 0.2rem; }

    strong { color: var(--text); font-weight: 600; }
    em { color: var(--yellow); font-style: normal; }

    /* ── LEAD / SUBTITLE ── */
    .lead {
      font-size: 1.05rem;
      color: var(--muted);
      margin-bottom: 2rem;
      line-height: 1.7;
    }

    /* ── PILL BADGES ── */
    .pill {
      display: inline-flex;
      align-items: center;
      gap: 0.25rem;
      padding: 0.15rem 0.5rem;
      border-radius: 999px;
      font-size: 0.65rem;
      font-weight: 600;
      border: 1px solid;
      vertical-align: middle;
    }
    .pill.green  { background: #00d4aa12; color: var(--green);  border-color: #00d4aa30; }
    .pill.purple { background: #6c63ff12; color: var(--accent); border-color: #6c63ff30; }
    .pill.yellow { background: #f5c54212; color: var(--yellow); border-color: #f5c54230; }
    .pill.red    { background: #ff5f5712; color: var(--red);    border-color: #ff5f5730; }

    /* ── CODE ── */
    code {
      font-family: 'SF Mono', 'Fira Code', monospace;
      font-size: 0.8em;
      background: var(--surface2);
      border: 1px solid var(--border);
      padding: 0.1em 0.4em;
      border-radius: 4px;
      color: #c8c8e0;
    }

    pre {
      background: var(--code-bg);
      border: 1px solid var(--border);
      border-radius: 10px;
      padding: 1.25rem 1.5rem;
      overflow-x: auto;
      margin: 1rem 0 1.5rem;
      font-family: 'SF Mono', 'Fira Code', monospace;
      font-size: 0.78rem;
      line-height: 1.65;
    }

    pre code {
      background: none;
      border: none;
      padding: 0;
      font-size: inherit;
      color: #c8c8e0;
    }

    /* syntax highlight classes */
    .kw  { color: #c792ea; }
    .fn  { color: #82aaff; }
    .str { color: #c3e88d; }
    .num { color: #f78c6c; }
    .cm  { color: #546e7a; font-style: italic; }
    .ty  { color: #ffcb6b; }
    .op  { color: #89ddff; }

    /* ── PRE HEADER (filename label) ── */
    .code-block { margin: 1rem 0 1.5rem; }
    .code-label {
      background: var(--surface2);
      border: 1px solid var(--border);
      border-bottom: none;
      border-radius: 8px 8px 0 0;
      padding: 0.4rem 1rem;
      font-family: 'SF Mono', 'Fira Code', monospace;
      font-size: 0.7rem;
      color: var(--muted);
      display: flex;
      align-items: center;
      gap: 0.5rem;
    }
    .code-block pre { margin: 0; border-radius: 0 0 8px 8px; }

    /* ── CALLOUTS ── */
    .callout {
      border-radius: 10px;
      padding: 1rem 1.25rem;
      margin: 1.25rem 0;
      font-size: 0.85rem;
      line-height: 1.65;
      border-left: 3px solid;
    }
    .callout p { margin: 0; color: inherit; font-size: inherit; }
    .callout.info    { background: #6c63ff0d; border-color: var(--accent); color: #a0a0d0; }
    .callout.tip     { background: #00d4aa0d; border-color: var(--green);  color: #80c8bc; }
    .callout.warning { background: #f5c5420d; border-color: var(--yellow); color: #c0a060; }
    .callout.danger  { background: #ff5f570d; border-color: var(--red);    color: #c08080; }

    .callout-title { font-weight: 700; margin-bottom: 0.3rem; color: inherit; }

    /* ── NARROW SCREENS ──
       This page shipped with NO media query at all, behind a 260px fixed
       sidebar. At 390px that left main a 130px border-box and, after its 3rem
       side padding, about 34px of readable text — with the sidebar covering
       two thirds of the screen. It is nav-linked from every page and sits in
       the sitemap, so a shared link opened on a phone landed on that. */
    @media (max-width: 900px) {
      :root { --sidebar-w: 0px; }
      aside {
        position: static; width: auto; min-width: 0; height: auto;
        border-right: 0; border-bottom: 1px solid var(--border);
        max-height: 40vh; overflow-y: auto;
      }
      main { margin-left: 0; padding: 2rem 1.1rem 4rem; max-width: none; }
      /* THE FLEX CONTEXT IS .layout, NOT body. The first version of this
         query reblocked body — which was already stacked — and left .layout
         a flex row, so the sidebar stayed a 124px side column of wrapped
         links beside ~230px of text. Reviewed at 390px; this is the line
         that actually stacks them. */
      .layout { display: block; }
      .topbar { padding: 0 0.8rem; gap: 0.8rem; }
    }

    /* ── TABLES ──
       Nineteen of them are direct children of main with no wrapper. The
       longest unbreakable cell is a weight name like
       transformer.h.N.post_attention_layernorm.weight, which cannot shrink,
       so on a phone the table pushed the BODY sideways rather than scrolling
       inside itself. Each table gets its own scroll container instead. */
    .table-scroll { overflow-x: auto; -webkit-overflow-scrolling: touch; margin: 1rem 0; }
    table {
      width: 100%;
      border-collapse: collapse;
      font-size: 0.82rem;
      margin: 1rem 0 1.5rem;
      border: 1px solid var(--border);
      border-radius: 8px;
      overflow: hidden;
    }
    th {
      background: var(--surface2);
      padding: 0.65rem 1rem;
      text-align: left;
      font-weight: 600;
      color: var(--text);
      border-bottom: 1px solid var(--border);
    }
    td {
      padding: 0.6rem 1rem;
      color: #b0b0bc;
      border-bottom: 1px solid var(--border);
      vertical-align: middle;
    }
    tr:last-child td { border-bottom: none; }
    tr:hover td { background: #ffffff03; }

    /* ── SHADER CARD ── */
    .shader-doc-card {
      background: var(--surface);
      border: 1px solid var(--border);
      border-radius: 12px;
      padding: 1.25rem 1.5rem;
      margin: 1.25rem 0;
    }

    .shader-doc-card h4 {
      font-size: 0.88rem;
      font-weight: 700;
      color: var(--text);
      text-transform: none;
      letter-spacing: 0;
      margin-top: 0;
      display: flex;
      align-items: center;
      gap: 0.65rem;
    }

    .shader-doc-card p { font-size: 0.82rem; color: var(--muted); margin: 0.4rem 0 0; }

    .binding-table { margin: 0.75rem 0 0; }
    .binding-table th, .binding-table td { padding: 0.4rem 0.75rem; font-size: 0.75rem; }

    /* ── STEP FLOW ── */
    .step-flow { display: flex; flex-direction: column; gap: 0; margin: 1rem 0 1.5rem; }

    .step-row {
      display: flex;
      gap: 1rem;
      align-items: flex-start;
    }

    .step-line {
      display: flex;
      flex-direction: column;
      align-items: center;
      flex-shrink: 0;
    }

    .step-dot {
      width: 28px; height: 28px;
      border-radius: 50%;
      background: var(--accent);
      color: #fff;
      font-size: 0.68rem;
      font-weight: 700;
      display: flex;
      align-items: center;
      justify-content: center;
      flex-shrink: 0;
    }

    .step-line-seg {
      width: 2px;
      flex: 1;
      background: var(--border);
      margin: 4px 0;
      min-height: 20px;
    }

    .step-content { padding: 0.05rem 0 1.25rem; }
    .step-content h5 { font-size: 0.88rem; font-weight: 700; color: var(--text); margin-bottom: 0.25rem; }
    .step-content p { font-size: 0.8rem; color: var(--muted); margin: 0; }

    /* ── DIVIDER ── */
    hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; }

    /* ── SCROLL MARGIN ── */
    [id] { scroll-margin-top: calc(var(--topbar-h) + 1.5rem); }
  </style>
</head>
<body>

<!-- ── TOPBAR ── -->
<div class="topbar">
  <a href="/" class="logo">Zero<span>TVM</span></a>
  <span class="topbar-sep">/</span>
  <span class="topbar-title">Documentation</span>
  <nav class="topbar-nav">
    <a href="/">Home</a>
    <a href="zero-tvm.html" class="topbar-cta">Open Chat →</a>
  </nav>
</div>

<div class="layout">

<!-- ── SIDEBAR ── -->
<aside>
  <div class="sidebar-section">
    <span class="sidebar-section-label">Getting Started</span>
    <a href="#intro" class="sidebar-link active">Introduction</a>
    <a href="#quickstart" class="sidebar-link">Quick start</a>
    <a href="#how-it-works" class="sidebar-link">How it works</a>
  </div>

  <div class="sidebar-section">
    <span class="sidebar-section-label">Architecture</span>
    <a href="#overview" class="sidebar-link">Overview</a>
    <a href="#weight-loader" class="sidebar-link">Weight loader</a>
    <a href="#tokenizer" class="sidebar-link">Tokenizer</a>
    <a href="#kv-cache" class="sidebar-link">KV cache</a>
    <a href="#decode-loop" class="sidebar-link">Decode loop</a>
    <a href="#ping-pong" class="sidebar-link">Ping-pong buffers</a>
  </div>

  <div class="sidebar-section">
    <span class="sidebar-section-label">WGSL Kernel Roles</span>
    <a href="#shaders" class="sidebar-link">Overview</a>
    <a href="#shader-embedding" class="sidebar-link sub">1 · Embedding</a>
    <a href="#shader-rmsnorm" class="sidebar-link sub">2 · RMSNorm</a>
    <a href="#shader-qkv" class="sidebar-link sub">3 · QKV+RoPE+KV (fused)</a>
    <a href="#shader-attention" class="sidebar-link sub">4 · Paged Attention</a>
    <a href="#shader-oproj" class="sidebar-link sub">5 · int4 Matmul</a>
    <a href="#shader-ffn" class="sidebar-link sub">6 · Fused FFN</a>
    <a href="#shader-addnorm" class="sidebar-link sub">7 · Add + RMSNorm</a>
    <a href="#shader-kvappend" class="sidebar-link sub">8 · KV Append (prefill)</a>
    <a href="#shader-rope" class="sidebar-link sub">9 · RoPE (prefill)</a>
    <a href="#shader-lmhead" class="sidebar-link sub">10 · Argmax</a>
  </div>

  <div class="sidebar-section">
    <span class="sidebar-section-label">Model</span>
    <a href="#phi3" class="sidebar-link">Phi-3 constants</a>
    <a href="#quantization" class="sidebar-link">Q4F16 format</a>
    <a href="#weight-loader" class="sidebar-link">Weight naming</a>
    <a href="#qwen3" class="sidebar-link">Qwen3-4B port</a>
    <a href="#qwen35" class="sidebar-link">Qwen3.5-4B hybrid</a>
<a href="#qwen36" class="sidebar-link">Qwen3.6 MoE</a>
  </div>

  <div class="sidebar-section">
    <span class="sidebar-section-label">Extending</span>
    <a href="#porting" class="sidebar-link">Port to Phi-4 / Qwen3</a>
    <a href="#local-weights" class="sidebar-link">Local weight serving</a>
    <a href="#debugging" class="sidebar-link">Debugging tips</a>
  </div>

  <div class="sidebar-section">
    <span class="sidebar-section-label">Reference</span>
    <a href="#bugs-fixed" class="sidebar-link">Bugs we fixed</a>
    <a href="#vs-webllm" class="sidebar-link">vs WebLLM</a>
  </div>
</aside>

<!-- ── MAIN ── -->
<main>

<!-- INTRO -->
<div id="intro">
  <h1>Zero TVM Documentation</h1>
  <p class="lead">
    A complete LLM inference engine in the browser — <strong>no WebLLM, no TVM, no ONNX, no WASM runtime</strong>.
    10 hand-written WGSL kernel roles, a BPE tokenizer, and raw WebGPU.
    Measured +16.0% on total wall-clock throughput and +31.4% on decode against WebLLM's TVM-autotuned
    kernels, identical weights, same session (M2 Max, 2026-07-30).
  </p>

  <div style="display:flex;gap:0.5rem;flex-wrap:wrap;margin-bottom:1.5rem;">
    <span class="pill green">WebGPU</span>
    <span class="pill purple">10 kernel roles</span>
    <span class="pill yellow">Q4F16_1</span>
    <span class="pill green">Phi-3-mini 3.8B</span>
    <span class="pill purple">Paged KV Cache</span>
    <span class="pill green">0 dependencies</span>
  </div>

  <p>
    This project implements the full Phi-3-mini-4k-instruct transformer forward pass using only
    WebGPU compute shaders written by hand in WGSL. Weights are loaded directly from HuggingFace
    in MLC Q4F16_1 format (cached in OPFS after the first load).
    The tokenizer is implemented in pure TypeScript — no SentencePiece WASM.
  </p>

  <div class="callout tip">
    <div class="callout-title">Why?</div>
    <p>WebLLM compiles its kernels with TVM; transformers.js runs ONNX Runtime Web. Both work well. Neither is something you read — this is.
    This project exists to show that you can understand <em>every single step</em> of a modern transformer
    inference pipeline, written at the GPU level, in a browser tab.</p>
  </div>
</div>

<!-- QUICKSTART -->
<h2 id="quickstart">Quick start</h2>

<p>No install. Just open the chat page. Weights load from your browser cache (if you've used WebLLM before) or download fresh from HuggingFace (~2 GB).</p>

<div class="step-flow">
  <div class="step-row">
    <div class="step-line"><div class="step-dot">1</div><div class="step-line-seg"></div></div>
    <div class="step-content">
      <h5>Open the chat</h5>
      <p>Navigate to <code>zero-tvm.html</code>. WebGPU initializes automatically. Requires WebGPU with shader-f16: Chrome 113+ or Edge 113+. Safari did not enable WebGPU by default until 26, and the writable OPFS API the weight cache needs (createWritable) is missing before then. Subgroups are required for the MoE models.</p>
    </div>
  </div>
  <div class="step-row">
    <div class="step-line"><div class="step-dot">2</div><div class="step-line-seg"></div></div>
    <div class="step-content">
      <h5>Wait for weights</h5>
      <p>First load downloads ~2 GB. Subsequent loads are instant from OPFS. On a small machine, <code>?ctx=</code> shrinks the context window and with it the KV allocation (~1.5 GB at Phi-3’s default 4K). Progress shown per shard in the log panel.</p>
    </div>
  </div>
  <div class="step-row">
    <div class="step-line"><div class="step-dot">3</div><div class="step-line-seg"></div></div>
    <div class="step-content">
      <h5>Chat</h5>
      <p>Phi-3-mini runs at 69.6 tok/s total (83.1 tok/s decode) on M2 Max — +16.0% / +31.4% vs WebLLM on identical weights, same session. Your conversation never leaves the browser. No API calls during inference.</p>
    </div>
  </div>
  <div class="step-row">
    <div class="step-line"><div class="step-dot">4</div></div>
    <div class="step-content">
      <h5>(Optional) Cache weights locally</h5>
      <p>Run <code>node scripts/download-weights.mjs</code> once to save the shards to <code>.weights-local/</code>. Subsequent loads are served from localhost at full disk speed.</p>
    </div>
  </div>
</div>

<div class="callout warning">
  <div class="callout-title">GPU requirement</div>
  <p>Requires a GPU with <code>shader-f16</code> WebGPU feature (f16 arithmetic). Most M-series Macs and recent NVIDIA/AMD GPUs support this. Intel integrated graphics may not.</p>
</div>

<!-- HOW IT WORKS -->
<h2 id="how-it-works">How it works</h2>

<p>The engine has three phases per generated token:</p>

<ol>
  <li><strong>Prefill</strong> — process the prompt in chunks (per token on specs that cannot chunk), building up the KV cache</li>
  <li><strong>First decode</strong> — the last prefill step produces the first generated token</li>
  <li><strong>Decode loop</strong> — each step takes the previous token as input, runs the full forward pass with the KV cache providing attention context, produces the next token</li>
</ol>

<p>Each forward pass runs <strong>10 kernel roles</strong> through 32 transformer layers — <strong>260 dispatches per token</strong> on the default path — split-K attention has been on by default since 2026-07-27 and adds a combine dispatch per layer; <code>?splitk=0</code> runs the 228-dispatch reference chain — then reads one i32 token ID back from the GPU. For comparison, WebLLM's TVM-generated decode path fires 342 dispatches per token — around 11 distinct shaders on that path, of 85 captured across a whole session.</p>

<!-- ARCHITECTURE -->
<h2 id="overview">Architecture overview</h2>

<p>The engine’s main source files:</p>

<div class="table-scroll"><table>
  <thead><tr><th>File</th><th>Role</th></tr></thead>
  <tbody>
    <tr><td><code>src/zero-tvm/chat.ts</code></td><td>Main decode engine + UI. Allocates buffers, builds bind groups, runs the decode loop.</td></tr>
    <tr><td><code>src/zero-tvm/weight-loader.ts</code></td><td>Fetches ndarray-cache.json, downloads shards, uploads to GPU buffers.</td></tr>
    <tr><td><code>src/zero-tvm/tokenizer.ts</code></td><td>BPE tokenizer: encode text → token IDs, decode IDs → text.</td></tr>
    <tr><td><code>src/compiler/compiler.ts</code></td><td>Compiles every shader — the hand-written WGSL files plus the generated int4-matmul variants (tiled/subgroup/affine/MoE) — into <code>GPUComputePipeline</code> objects.</td></tr>
  </tbody>
</table></div>

<!-- WEIGHT LOADER -->
<h2 id="weight-loader">Weight loader</h2>

<p>
  Weights are stored in MLC's <strong>ndarray-cache.json</strong> format — an index file listing every parameter,
  which shard binary it lives in, its byte offset, and byte size.
  The loader reads this index then fetches each referenced shard.
</p>

<h3>Fetch priority</h3>
<ol>
  <li><strong>OPFS</strong> — where this engine caches; a returning visitor loads from here</li>
  <li><strong>Browser Cache API</strong> — read-only leftover from prior WebLLM sessions</li>
  <li><strong>HuggingFace</strong> — direct HTTPS fetch from <code>huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC</code></li>
</ol>

<div class="callout danger">
  <div class="callout-title">Version mismatch pitfall</div>
  <p>Do not mix a locally-downloaded <code>ndarray-cache.json</code> with shards from the browser cache.
  The byte offsets in the index must match the shards exactly. If you download the index fresh but use
  cached shards from an older model version, all weight slices will be wrong → zero logits → <code>&lt;unk&gt;</code> output.</p>
</div>

<h3>Parameter naming (MLC format)</h3>
<p>MLC uses non-standard parameter names. The actual names in the cache:</p>

<div class="table-scroll"><table>
  <thead><tr><th>MLC name</th><th>Role</th></tr></thead>
  <tbody>
    <tr><td><code>transformer.embd.q_weight</code></td><td>Embedding weights (uint32 packed int4)</td></tr>
    <tr><td><code>transformer.embd.q_scale</code></td><td>Embedding scales (f16)</td></tr>
    <tr><td><code>transformer.norm.weight</code></td><td>Final RMSNorm gamma (after all layers)</td></tr>
    <tr><td><code>lm_head.q_weight</code></td><td>LM head weights</td></tr>
    <tr><td><code>transformer.h.N.ln.weight</code></td><td>Layer N input_layernorm (normGamma1)</td></tr>
    <tr><td><code>transformer.h.N.post_attention_layernorm.weight</code></td><td>Layer N post-attention norm (normGamma2)</td></tr>
    <tr><td><code>transformer.h.N.mixer.qkv_proj.q_weight</code></td><td>Layer N QKV projection weights</td></tr>
    <tr><td><code>transformer.h.N.mixer.out_proj.q_weight</code></td><td>Layer N output projection weights</td></tr>
    <tr><td><code>transformer.h.N.mlp.gate_up_proj.q_weight</code></td><td>Layer N FFN gate+up weights</td></tr>
    <tr><td><code>transformer.h.N.mlp.down_proj.q_weight</code></td><td>Layer N FFN down weights</td></tr>
  </tbody>
</table></div>

<!-- TOKENIZER -->
<h2 id="tokenizer">Tokenizer</h2>

<p>
  A hand-written BPE tokenizer in TypeScript. No SentencePiece WASM, no HuggingFace tokenizers bundle.
  Reads <code>tokenizer.json</code> directly.
</p>

<h3>Key steps</h3>
<ul>
  <li><strong>Pre-tokenization</strong> — Metaspace: spaces become <code>▁</code>, words are split on whitespace</li>
  <li><strong>BPE encoding</strong> — merge pairs by rank from the merge table in tokenizer.json</li>
  <li><strong>Special tokens</strong> — <code>&lt;|system|&gt;</code>, <code>&lt;|user|&gt;</code>, <code>&lt;|assistant|&gt;</code>, <code>&lt;|end|&gt;</code></li>
  <li><strong>Chat template</strong> — Phi-3 format applied by <code>buildChatPrompt()</code></li>
</ul>

<h3>Phi-3 chat template</h3>
<div class="code-block">
  <div class="code-label">📄 prompt format</div>
  <pre><code>&lt;|system|&gt;
You are a helpful assistant.&lt;|end|&gt;
&lt;|user|&gt;
What is the capital of Australia?&lt;|end|&gt;
&lt;|assistant|&gt;</code></pre>
</div>

<p>Stop tokens: <code>2</code> (EOS), <code>32000</code> (<code>&lt;|end|&gt;</code>), <code>32007</code> (<code>&lt;|endoftext|&gt;</code>).</p>

<!-- KV CACHE -->
<h2 id="kv-cache">KV cache</h2>

<p>
  Uses a <strong>paged KV cache</strong> layout. Memory is divided into fixed-size pages (16 slots each)
  and a page table maps logical positions to physical pages. This is <em>not</em> vLLM-style relocatable
  blocks: K is RoPE&rsquo;d before the cache write on every path, so a cached page is valid only at the
  positions it was written at. It is a prefix pool, and cannot become a block pool without moving RoPE.
</p>

<div class="table-scroll"><table>
  <thead><tr><th>Parameter</th><th>Value</th><th>Notes</th></tr></thead>
  <tbody>
    <tr><td><code>PAGE_SIZE</code></td><td>16</td><td>slots per page</td></tr>
    <tr><td><code>MAX_PAGES</code></td><td>257</td><td>≈ 4096 context tokens</td></tr>
    <tr><td>Bytes per page</td><td>196,608</td><td>32 heads × 16 slots × 96 dims × 2 (K+V) × 2 bytes</td></tr>
    <tr><td>Total KV buffer</td><td>~50 MB per layer</td><td>32 layers = ~1.6 GB</td></tr>
  </tbody>
</table></div>

<p>Each layer has its own <code>GPUBuffer</code> for KV pages. The page table is a simple identity mapping (page i → physical page i) for single-sequence inference.</p>

<!-- DECODE LOOP -->
<h2 id="decode-loop">Decode loop</h2>

<p>Each call to <code>decodeToken(tokenId, position)</code> submits one command encoder with the full forward pass:</p>

<div class="code-block">
  <div class="code-label">src/zero-tvm/chat.ts · decodeToken()</div>
  <pre><code><span class="cm">// Per-token GPU state written via writeBuffer</span>
B.inputIds   ← [tokenId]          <span class="cm">// i32</span>
B.posMap     ← [position]         <span class="cm">// i32</span>
B.pageIndptr ← [0, nnzPages]      <span class="cm">// page range</span>
B.lengthInfo ← [position+1, 0, 0] <span class="cm">// seq length</span>

<span class="cm">// Forward pass (one command encoder)</span>
embedding(B.residual)             <span class="cm">// token → hidden state</span>
rmsNorm(B.hidden1, B.residual)    <span class="cm">// initial norm</span>

<span class="kw">for</span> L <span class="op">in</span> <span class="num">0</span>..<span class="num">32</span>:
  <span class="cm">// QKV matmul + RoPE + KV-append in ONE dispatch (M4 fusion)</span>
  qkvFused(B.qOut, kvPages[L], B.hidden1)
  attention(B.attnOut, B.qOut, kvPages[L])
  int4Matmul(B.hidden2, B.attnOut)               <span class="cm">// O projection</span>
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  <span class="cm">// residual + RMSNorm, ping-pong</span>
  <span class="cm">// Gate + Up + SiLU + mul + Down in ONE dispatch</span>
  fusedFfn(B.hidden2, B.hidden1)
  addNorm(B.hidden2, resIn → B.hidden1, resOut)  <span class="cm">// residual + RMSNorm, ping-pong</span>

int4Matmul(B.logits, B.hidden1)   <span class="cm">// LM head</span>
argmax(B.tokenOut, B.logits)      <span class="cm">// → next token ID</span></code></pre>
</div>

<!-- PING-PONG -->
<h2 id="ping-pong">Ping-pong residual buffers</h2>

<p>
  WebGPU's validation rules forbid binding the same buffer as both <code>read</code> and <code>read_write</code>
  in the same dispatch. The add_norm shader needs to <em>read</em> the old residual and <em>write</em> the new one.
</p>

<p>Solution: two residual buffers that alternate each dispatch.</p>

<div class="code-block">
  <div class="code-label">ping-pong pattern</div>
  <pre><code><span class="kw">let</span> resIn  = B.residual   <span class="cm">// ping (starts with embedding)</span>
<span class="kw">let</span> resOut = B.residual2  <span class="cm">// pong (uninitialized)</span>

<span class="cm">// Each add_norm:</span>
dispatch(addNorm, [delta, resIn, gamma, hidden1, resOut])
[resIn, resOut] = [resOut, resIn]  <span class="cm">// swap — O(1), no GPU copy</span></code></pre>
</div>

<div class="callout info">
  <p>The swap is just two JavaScript variable reassignments — no GPU buffer copy. Both buffers always exist on the GPU; we just change which one we tell the bind group to read vs write.</p>
</div>

<!-- SHADERS -->
<h2 id="shaders">WGSL Kernel Roles</h2>

<p>All shaders live in <code>src/compiler/shaders/</code>, implementing 10 distinct kernel roles. The rest are tiled and subgroup variants of the same role, selectable at runtime via URL flags. The compiler compiles them all at startup into <code>GPUComputePipeline</code> objects.</p>

<p>Binding convention: <code>@group(0)</code> always. Binding indices are <strong>zero-based</strong> and match the order you pass buffers to <code>bg(device, pipeline, [...bufs])</code>.</p>

<div class="callout tip">
  <div class="callout-title">Decode vs prefill paths diverge</div>
  <p>The decode loop uses the fused <code>qkv_fused</code> kernel (QKV matmul + RoPE + KV-append all in one dispatch). Prefill still uses separate <code>int4_matmul</code> + <code>rope</code> + <code>kv_append</code> dispatches because prefill processes many tokens at once, and the fusion win only lands for <code>ntoken=1</code>.</p>
</div>

<hr>

<!-- SHADER 1 -->
<div id="shader-embedding">
  <div class="shader-doc-card">
    <h4>🔤 1 · Embedding <span class="pill purple">embedding.wgsl</span></h4>
    <p>Token ID lookup with Q4F16 dequantization. Each output element is dequantized from a packed int4 value: <code>(nibble - 7) × scale</code>.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>output hidden state</td></tr>
        <tr><td><code>@1</code></td><td><code>read i32[]</code></td><td>input token IDs</td></tr>
        <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>scales (group_size=32)</td></tr>
        <tr><td><code>@3</code></td><td><code>read u32[]</code></td><td>packed weights (8 int4 per u32)</td></tr>
        <tr><td><code>@4</code></td><td><code>uniform</code></td><td><code>{ seq_len, packGridDimX }</code></td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 12 workgroups × 256 threads = 3072 output elements (D=3072)</p>
  </div>
</div>

<!-- SHADER 2 -->
<div id="shader-rmsnorm">
  <div class="shader-doc-card">
    <h4>📐 2 · RMSNorm <span class="pill purple">rms_norm.wgsl</span></h4>
    <p>Root mean square layer normalization. Computes <code>x / sqrt(mean(x²) + ε) × gamma</code>. Uses 256-thread tree reduction in workgroup shared memory.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>normalized output</td></tr>
        <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>input</td></tr>
        <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gamma weights</td></tr>
        <tr><td><code>@3</code></td><td><code>uniform</code></td><td><code>{ packGridDimX }</code></td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup (one token, D=3072)</p>
  </div>
</div>

<!-- SHADER 3 · QKV FUSED (decode) -->
<div id="shader-qkv">
  <div class="shader-doc-card">
    <h4>⚡ 3 · QKV + RoPE + KV-append (fused, decode) <span class="pill purple">qkv_fused.wgsl</span></h4>
    <p>
      The big M4 fusion. One dispatch replaces three on the decode path: the int4 QKV matmul, the RoPE rotation of Q and K, and the write of K/V into the paged KV cache. Each workgroup computes two output rows that form a RoPE pair (dim and dim+48 within the same head), rotates the pair in registers, and writes K/V straight into <code>kv_pages</code> — the intermediate <code>qkv</code> / <code>k_out</code> / <code>v_out</code> buffers from the pre-fusion path are skipped entirely.
    </p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>q_out [3072]</td></tr>
        <tr><td><code>@1</code></td><td><code>read_write f16[]</code></td><td>kv_pages (paged KV cache)</td></tr>
        <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>hidden [3072]</td></tr>
        <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>scales [9216 × 96]</td></tr>
        <tr><td><code>@4</code></td><td><code>read u32[]</code></td><td>packed weights [9216 × 384]</td></tr>
        <tr><td><code>@5</code></td><td><code>read i32[]</code></td><td>position map</td></tr>
        <tr><td><code>@6</code></td><td><code>uniform</code></td><td><code>{ position_map_elem_offset, pages_elem_offset, packGridDimX }</code></td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 4,608 workgroups (down from 9,216 matmul + 36 RoPE + 12 KV-append = 9,264 in the pre-fusion path). Decode-only; prefill still uses the 3-dispatch path (see shaders 8 and 9).</p>
  </div>
</div>

<!-- SHADER 4 · PAGED ATTENTION -->
<div id="shader-attention">
  <div class="shader-doc-card">
    <h4>👁️ 4 · Paged Attention <span class="pill purple">attention.wgsl</span></h4>
    <p>Multi-head attention over the paged KV cache. Reads K and V from pages, computes scaled dot-product attention with an online-softmax reduction in shared memory. Each workgroup handles one attention head.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>Q [3072]</td></tr>
        <tr><td><code>@1</code></td><td><code>read i32[]</code></td><td>page indptr</td></tr>
        <tr><td><code>@2</code></td><td><code>read i32[]</code></td><td>page values (page table)</td></tr>
        <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>KV pages</td></tr>
        <tr><td><code>@4</code></td><td><code>read i32[]</code></td><td>length info</td></tr>
        <tr><td><code>@5</code></td><td><code>read_write f16[]</code></td><td>attn output [3072]</td></tr>
        <tr><td><code>@6</code></td><td><code>uniform</code></td><td>attention config (scale, pages)</td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 × HEADS workgroups (1 × 32). An <code>attention_int8.wgsl</code> variant reads an int8-quantized KV cache; enable via <code>?kv8=1</code>.</p>
  </div>
</div>

<!-- SHADER 5 · INT4 MATMUL -->
<div id="shader-oproj">
  <div class="shader-doc-card">
    <h4>✖️ 5 · int4 Matmul (output projection + LM head) <span class="pill purple">int4_matmul.wgsl</span></h4>
    <p>General-purpose dequantize-on-the-fly int4 × f16 matmul. Used for the attention <em>output projection</em> (3072 → 3072) and the <em>LM head</em> (3072 → 32064). Weights are Q4F16_1: <code>N</code> output rows × 384 u32 columns (each u32 = 8 int4 values = 32 elements, group_size=32). The tiled / subgroup / vec4 variants are emitted by one generator beside it (<code>int4_matmul.gen.ts</code>); the runtime picks one via the <code>?matmul=</code> URL flag.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Uniform field</th><th>Value</th></tr></thead>
      <tbody>
        <tr><td><code>K_groups</code></td><td>384 (= input_dim / 8)</td></tr>
        <tr><td><code>scale_stride</code></td><td>96 (= input_dim / group_size)</td></tr>
        <tr><td><code>N</code></td><td>3072 (o-proj) or 32064 (lm_head)</td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> <code>N</code> workgroups — 3,072 for o-proj, 32,064 for lm_head.</p>
  </div>
</div>

<!-- SHADER 6 · FUSED FFN -->
<div id="shader-ffn">
  <div class="shader-doc-card">
    <h4>🔀 6 · Fused FFN (Gate · Up · SiLU · Mul · Down) <span class="pill purple">fused_ffn.wgsl</span></h4>
    <p>
      Gate and up projections (both int4 matmuls sharing the 16,384-row <code>gate_up_proj</code> weight matrix), the SiLU activation and the elementwise multiply, in one dispatch. The down projection back to 3,072 dims is its own <code>int4_matmul</code> dispatch — which is why a layer costs 7 dispatches, not 6.
    </p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>output [3072]</td></tr>
        <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>input [3072]</td></tr>
        <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gate_up scales</td></tr>
        <tr><td><code>@3</code></td><td><code>read u32[]</code></td><td>gate_up packed weights (16,384 × 384)</td></tr>
        <tr><td><code>@4</code></td><td><code>read f16[]</code></td><td>down_proj scales</td></tr>
        <tr><td><code>@5</code></td><td><code>read u32[]</code></td><td>down_proj packed weights (3,072 × 1,024)</td></tr>
        <tr><td><code>@6</code></td><td><code>uniform</code></td><td>FFN config</td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 3,072 workgroups — one per output row of the down projection. A <code>fused_ffn_tiled_sg.wgsl</code> variant uses subgroup reductions; selectable via URL flag.</p>
  </div>
</div>

<!-- SHADER 7 · ADD + RMSNORM -->
<div id="shader-addnorm">
  <div class="shader-doc-card">
    <h4>➕ 7 · Fused Add + RMSNorm <span class="pill purple">add_norm.wgsl</span></h4>
    <p>Residual add + RMSNorm in one pass. Computes <code>residual_out = A + B</code>, then <code>output = RMSNorm(residual_out) × gamma</code>. Used twice per layer (post-attention and post-FFN). Mirrors TVM's <code>fuse_add_norm_decode</code>.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>A — the new contribution (O-proj or FFN-down output)</td></tr>
        <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>B — the running residual (resIn)</td></tr>
        <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gamma — normalization weights</td></tr>
        <tr><td><code>@3</code></td><td><code>read_write f16[]</code></td><td>normalized output (B.hidden1)</td></tr>
        <tr><td><code>@4</code></td><td><code>read_write f16[]</code></td><td>new residual (resOut — ping-pong)</td></tr>
        <tr><td><code>@5</code></td><td><code>uniform</code></td><td><code>{ packGridDimX }</code></td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup · 256 threads · 12 elements each = 3,072.</p>
  </div>
</div>

<!-- SHADER 8 · KV APPEND (prefill) -->
<div id="shader-kvappend">
  <div class="shader-doc-card">
    <h4>💾 8 · KV Append (prefill path) <span class="pill purple">kv_append.wgsl</span></h4>
    <p>Writes K and V vectors into the paged KV cache at the correct slot for each position. On the decode path this work is folded into <code>qkv_fused</code>; on prefill it runs as a separate dispatch because prefill processes many tokens at once and the per-token fusion no longer pays off.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>k_out [3072]</td></tr>
        <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>v_out [3072]</td></tr>
        <tr><td><code>@2</code></td><td><code>read_write f16[]</code></td><td>KV pages buffer</td></tr>
        <tr><td><code>@3</code></td><td><code>read i32[]</code></td><td>position map</td></tr>
        <tr><td><code>@4</code></td><td><code>uniform</code></td><td>page config</td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 12 workgroups per token (HEADS=32, HEAD_DIM=96).</p>
  </div>
</div>

<!-- SHADER 9 · ROPE (prefill) -->
<div id="shader-rope">
  <div class="shader-doc-card">
    <h4>🌀 9 · RoPE (prefill path) <span class="pill purple">rope.wgsl</span></h4>
    <p>Rotary position embeddings applied to Q and K. Prefill-only — the decode path folds RoPE into <code>qkv_fused</code>. Splits the concatenated 9,216-dim QKV buffer into Q / K / V, rotates Q and K in place based on position, and copies V unchanged.</p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>q_out [3072]</td></tr>
        <tr><td><code>@1</code></td><td><code>read_write f16[]</code></td><td>k_out [3072]</td></tr>
        <tr><td><code>@2</code></td><td><code>read_write f16[]</code></td><td>v_out [3072]</td></tr>
        <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>qkv input [9216]</td></tr>
        <tr><td><code>@4</code></td><td><code>read i32[]</code></td><td>position map</td></tr>
        <tr><td><code>@5</code></td><td><code>uniform</code></td><td>RoPE config</td></tr>
      </tbody>
    </table></div>
    <div class="callout danger" style="margin-top:0.75rem;">
      <div class="callout-title">Critical binding order</div>
      <p>The binding order <strong>must</strong> be <code>[q_out, k_out, v_out, qkv, posMap, uniform]</code>. Swapping these caused a garbage-output bug during development.</p>
    </div>
    <p><strong>Dispatch:</strong> 36 workgroups × 256 threads = 9,216 = 3 × 3,072.</p>
  </div>
</div>

<!-- SHADER 10 · ARGMAX -->
<div id="shader-lmhead">
  <div class="shader-doc-card">
    <h4>🎯 10 · Argmax Sampler <span class="pill purple">argmax.wgsl</span></h4>
    <p>
      Parallel-reduction argmax over the 32,064-entry logit buffer produced by the LM-head <code>int4_matmul</code>. Replaces TVM's ~20-dispatch sampling chain (penalty → softmax → cumsum → argsort → gather → …) with a single dispatch. Greedy decoding only; top-k / top-p not wired up yet.
    </p>
    <div class="table-scroll"><table class="binding-table">
      <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead>
      <tbody>
        <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>logits [32064]</td></tr>
        <tr><td><code>@1</code></td><td><code>read_write i32[]</code></td><td>output token id [1]</td></tr>
      </tbody>
    </table></div>
    <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup (tree reduction over 32,064 logits). An <code>argmax_sg.wgsl</code> subgroup variant is available.</p>
  </div>
</div>

<!-- PHI-3 CONSTANTS -->
<h2 id="phi3">Phi-3 model constants</h2>

<div class="code-block">
  <div class="code-label">src/compiler/compiler.ts</div>
  <pre><code><span class="kw">export const</span> PHI3 = {
  D:        <span class="num">3072</span>,   <span class="cm">// hidden dimension</span>
  HEADS:    <span class="num">32</span>,     <span class="cm">// attention heads</span>
  HEAD_DIM: <span class="num">96</span>,     <span class="cm">// D / HEADS</span>
  LAYERS:   <span class="num">32</span>,     <span class="cm">// transformer layers</span>
  FFN:      <span class="num">8192</span>,   <span class="cm">// FFN intermediate dimension</span>
  VOCAB:    <span class="num">32064</span>,  <span class="cm">// vocabulary size</span>
  PAGE_SIZE:<span class="num">16</span>,     <span class="cm">// KV cache slots per page</span>
  MAX_PAGES:<span class="num">257</span>,    <span class="cm">// max pages (≈ 4096 context)</span>
}</code></pre>
</div>

<!-- Q4F16 -->
<h2 id="quantization">Q4F16 quantization format</h2>

<p>
  MLC's Q4F16_1 format packs 8 int4 values into each <code>uint32</code>.
  Scales are stored as <code>float16</code> with <code>group_size=32</code> — one scale per 32 weights.
</p>

<div class="code-block">
  <div class="code-label">Dequantization formula (from embedding.wgsl)</div>
  <pre><code><span class="cm">// Extract nibble for element i within a u32</span>
<span class="kw">let</span> nibble = (packed_u32 >> (i * <span class="num">4</span>)) & <span class="num">0xF</span>;

<span class="cm">// Dequantize: center around 0, multiply by scale</span>
<span class="kw">let</span> value = <span class="ty">f16</span>(i32(nibble) - <span class="num">7</span>) * scale;</code></pre>
</div>

<p>Weight shapes in Q4F16 (for Phi-3-mini):</p>

<div class="table-scroll"><table>
  <thead><tr><th>Parameter</th><th>q_weight shape (u32)</th><th>q_scale shape (f16)</th></tr></thead>
  <tbody>
    <tr><td>Embedding</td><td>[32064, 384]</td><td>[32064, 96]</td></tr>
    <tr><td>QKV proj (per layer)</td><td>[9216, 384]</td><td>[9216, 96]</td></tr>
    <tr><td>O proj (per layer)</td><td>[3072, 384]</td><td>[3072, 96]</td></tr>
    <tr><td>Gate+Up FFN (per layer)</td><td>[16384, 384]</td><td>[16384, 96]</td></tr>
    <tr><td>Down FFN (per layer)</td><td>[3072, 1024]</td><td>[3072, 256]</td></tr>
    <tr><td>LM head</td><td>[32064, 384]</td><td>[32064, 96]</td></tr>
  </tbody>
</table></div>

<!-- QWEN3 -->
<h2 id="qwen3">Qwen3-4B (<code>?model=qwen3</code>)</h2>

<p>
  The engine is parameterized over a <code>ModelSpec</code>, and a v1 Qwen3-4B (q4f16_1) port ships alongside Phi-3.
  Append <code>?model=qwen3</code> to <code>zero-tvm.html</code> or <code>validate.html</code> — it works on the live
  site (weights stream from HuggingFace, ~2.3 GB), and <code>node scripts/download-weights.mjs --model qwen3</code>
  primes the local dev mirror. Phi-3 stays the default; all existing URLs keep their exact behavior.
</p>

<p>What the port exercises that Phi-3 doesn't:</p>

<div class="table-scroll"><table>
  <thead><tr><th></th><th>Phi-3-mini (default)</th><th>Qwen3-4B</th></tr></thead>
  <tbody>
    <tr><td>Attention</td><td>MHA, 32/32 heads</td><td>GQA, 32 query heads over 8 KV heads</td></tr>
    <tr><td>QK-norm</td><td>none</td><td>per-head RMSNorm on Q and K between projection and RoPE</td></tr>
    <tr><td>Tokenizer</td><td>SentencePiece</td><td>byte-level BPE (Qwen2-style <code>tokenizer.json</code>)</td></tr>
    <tr><td>LM head</td><td>separate</td><td>tied — logits reuse the quantized embedding matrix (151,936 vocab)</td></tr>
    <tr><td>Decode path</td><td>fused, 7 dispatches/layer</td><td>unfused QKV + fused qk_norm+RoPE+append, 8 dispatches/layer</td></tr>
  </tbody>
</table></div>

<p>
  Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight
  bytes, both engines paying a full prefill on every run): Zero-TVM <strong>59.85 tok/s total</strong>
  (TTFT 453 ms, decode 75.49) vs WebLLM 0.2.84's prebuilt Qwen3-4B at <strong>45.46 tok/s total</strong>
  (self-reported decode 47.77) — <strong>+31.7% total, +58.0% decode</strong>. Unlike the Phi-3 headline,
  these figures are static dated text, not synced from <code>bench/results.json</code>; the machine-readable
  record is <code>bench/results/qwen3-4b.json</code>.
</p>

<div class="callout warning">
  <div class="callout-title">Read it cautiously — one pair, one machine, and the previous number was withdrawn</div>
  <p>The <strong>"75.74 vs 43.75, +73%" pair published on 2026-07-29 is withdrawn.</strong> It was measured after cross-turn prefix reuse shipped (PR #24) but before the bench harness was fixed to reset it, so the Zero-TVM half prefilled a single token per run while the WebLLM half prefilled the whole prompt — not like-for-like. Full writeup at the top of BENCH.md. The engine work itself is unchanged and its Zero-TVM-vs-Zero-TVM A/Bs still stand: QK-norm must run between the QKV matmul and RoPE, which is incompatible with the fused QKV+RoPE+KV-append kernel — the QKV matmul stays a separate dispatch — but since the 2026-07-29 tuning round everything after it is fused (<code>qk_norm_rope_append</code>: per-head norm + RoPE + paged KV write in one pass, 8 dispatches/layer; <code>?fuseqk=0</code> restores the 10-dispatch reference chain) and the K%512 <code>_vec4h</code> matmul variants give d=2560 / ffn=9728 wide loads (<code>?vec4h=0</code> opts out). Same-day A/Bs: fused-qk +2.3%, vec4h +5.7%, combined +5.8% over the flags-off half. The earlier 2026-07-28 pair (25.43 vs 14.15) did not reproduce on the same machine — both engines moved ~3× together (degraded session; control-run details in BENCH.md's tuning-round session note). Qwen3-4B is also the model where WebLLM most clearly beats us on time-to-first-token: its 263–271 tok/s prefill implies ~150 ms against our 453 ms.</p>
</div>

<!-- QWEN35 -->
<h2 id="qwen35">Qwen3.5-4B hybrid (<code>?model=qwen35</code>)</h2>

<p>
  The third model, and the first <em>hybrid</em> architecture on the engine: 24 gated-DeltaNet (linear-attention)
  layers interleaved with 8 gated full-attention layers (attention on every 4th layer). To our knowledge this is
  the first hand-written-kernel int4 implementation of a gated-DeltaNet hybrid running in a browser. Append
  <code>?model=qwen35</code> to <code>zero-tvm.html</code> or <code>validate.html</code>;
  <code>node scripts/download-weights.mjs --model qwen35</code> primes the local dev mirror (~2.6 GB).
</p>

<p>What the hybrid adds over Qwen3-4B:</p>

<div class="table-scroll"><table>
  <thead><tr><th></th><th>Qwen3-4B</th><th>Qwen3.5-4B</th></tr></thead>
  <tbody>
    <tr><td>Layer stack</td><td>36 × attention</td><td>24 × gated DeltaNet + 8 × gated attention (every 4th layer)</td></tr>
    <tr><td>Sequence mixer (most layers)</td><td>GQA attention + KV cache</td><td>delta-rule recurrent state (16 k-heads / 32 v-heads, head dims 128, short conv K=4) — no KV cache on those layers</td></tr>
    <tr><td>Attention layers</td><td>GQA 32/8, head_dim 128, full RoPE</td><td>GQA 16/4, head_dim 256, partial RoPE (64 of 256 dims), sigmoid output gate per head</td></tr>
    <tr><td>Vocab</td><td>151,936</td><td>248,320 (renumbered specials — the shipped <code>mlc-chat-config.json</code> still lists stale Qwen3 stop ids; stops resolve from <code>tokenizer.json</code>)</td></tr>
    <tr><td>Weight manifest</td><td><code>ndarray-cache.json</code></td><td><code>tensor-cache.json</code> (MLC renamed it)</td></tr>
  </tbody>
</table></div>

<p>
  Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight
  bytes, both engines paying a full prefill on every run): Zero-TVM <strong>65.28 tok/s total</strong>
  (TTFT 171 ms, decode 73.30) vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at <strong>32.56 tok/s total</strong>
  (self-reported decode 34.32) — <strong>+100.5% total, +113.6% decode</strong>. This is the one model where
  first-token latency is roughly a wash rather than a loss (our 171 ms against an implied ~0.2 s from
  WebLLM's 175–177 tok/s prefill). Static dated text, not synced from <code>bench/results.json</code>;
  the machine-readable record is <code>bench/results/qwen35-4b.json</code>.
</p>

<div class="callout warning">
  <div class="callout-title">Read it cautiously — the GDN kernels are still scalar, so decode remains a floor, and the previous number was withdrawn</div>
  <p>The <strong>"65.67 vs 34.04, +93%" cross-check published on 2026-07-29 is withdrawn</strong> — same defect as the Qwen3 pair: measured after cross-turn prefix reuse shipped and before the bench harness reset it, so only the WebLLM half was paying prefill. The two earlier pairs (53.07 vs 32.36, +64%, from the hybrid perf round; 47.99 vs 31.99, +50%, the v1 floor) predate prefix reuse and were like-for-like — superseded, not defective. Full writeup at the top of BENCH.md. Engine caveats unchanged: the 24 DeltaNet layers run scalar (non-subgroup) kernels, so the decode number is a floor; the input projections are fused 4→1 per GDN layer and prompts prefill in chunks of ≤64 since the 2026-07-29 prefill round, but the rest of the Phi-3 fusion story has not been applied to the GDN half. One machine, one pair; full protocol and caveats in BENCH.md.</p>
</div>

<!-- PORTING -->
<h2 id="qwen36">Qwen3.6-35B-A3B MoE (<code>?model=qwen36q3</code> / <code>?model=qwen36</code>)</h2>
<p>
  The fourth model, shipped 2026-08-05, and three firsts at once: the first sparse MoE
  (256 experts, top-8 plus a shared expert on every layer), the first MLX-format
  checkpoint (affine <code>w&nbsp;=&nbsp;s·q&nbsp;+&nbsp;b</code>, group 64, per-tensor
  biases, loaded by byte range — a 5.3&nbsp;GB safetensors shard is never one
  <code>ArrayBuffer</code>), and the first model here with <strong>no WebLLM build to
  benchmark against</strong>. <code>?model=qwen36q3</code> is the 3-bit-expert build
  (~16.4&nbsp;GB, needs ~20&nbsp;GB free RAM, ~66&nbsp;tok/s on a quiet 32&nbsp;GB
  M2&nbsp;Max); <code>?model=qwen36</code> is full 4-bit (~19.5&nbsp;GB, needs
  ~24&nbsp;GB free RAM). The MoE block runs in 7 dispatches with the expert index in
  grid&nbsp;<code>z</code>; every layer is validated against mlx_lm's own modules
  (<code>npm run test:kernels:real</code>). Full engineering notes live in the repo:
  <a href="https://github.com/abgnydn/zero-tvm#readme">README</a>,
  <a href="https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md">BENCH.md</a>,
  <a href="https://github.com/abgnydn/zero-tvm/blob/main/CLAUDE.md">CLAUDE.md</a>.
</p>

<h2 id="porting">Port to Phi-4-mini or Qwen3</h2>

<p>Both are available as MLC Q4F16 packages. The Qwen3 port has since landed (see <a href="#qwen3">Qwen3-4B</a> above) — these are the steps it followed, and the recipe for the next model:</p>

<ol>
  <li><strong>Update <code>PHI3</code> constants</strong> in <code>compiler.ts</code> — D, HEADS, HEAD_DIM, LAYERS, FFN, VOCAB</li>
  <li><strong>Check parameter names</strong> — fetch <code>ndarray-cache.json</code> and log all keys. Update <code>weight-loader.ts</code> candidates to match</li>
  <li><strong>Check for GQA</strong> — if KV heads ≠ Q heads (grouped-query attention), the attention shader needs a small change to repeat KV heads</li>
  <li><strong>Update chat template</strong> — each model has its own special tokens and prompt format</li>
  <li><strong>Update HuggingFace base URL</strong> — change <code>PHI3_MODEL_BASE</code> in <code>weight-loader.ts</code></li>
</ol>

<div class="callout tip">
  <div class="callout-title">Phi-4-mini is the easiest port</div>
  <p>Same family as Phi-3. MLC package is already available at <code>mlc-ai/Phi-4-mini-instruct-q4f16_1-MLC</code>. Parameter naming is likely identical or very similar.</p>
</div>

<!-- LOCAL WEIGHTS -->
<h2 id="local-weights">Local weight serving</h2>

<p>Run the download script once to save all shards locally. Subsequent page loads are instant (served from localhost, no network).</p>

<div class="code-block">
  <div class="code-label">terminal</div>
  <pre><code>node scripts/download-weights.mjs

<span class="cm"># Downloads to: .weights-local/Phi-3-mini-4k-instruct-q4f16_1-MLC/</span>
<span class="cm"># Served at:    /local-weights/Phi-3-mini-4k-instruct-q4f16_1-MLC/</span>
<span class="cm"># Size:         ~2 GB</span></code></pre>
</div>

<div class="callout danger">
  <div class="callout-title">Keep index + shards in sync</div>
  <p>Always download everything together. Never mix a freshly-downloaded <code>ndarray-cache.json</code> with old cached shards — the byte offsets will not match and all weights will be corrupted.</p>
</div>

<!-- DEBUGGING -->
<h2 id="debugging">Debugging tips</h2>

<h3>All output is <code>&lt;unk&gt;</code></h3>
<ul>
  <li>Weight version mismatch — <code>ndarray-cache.json</code> offsets don't match shard content</li>
  <li>Buffer aliasing — <code>add_norm</code> dispatched with same buffer as both <code>@1</code> and <code>@4</code></li>
  <li>Wrong rope binding order — check <code>@0=q_out, @1=k_out, @2=v_out, @3=qkv, @4=posMap</code></li>
</ul>

<h3>Garbage / repetitive output</h3>
<ul>
  <li>Rope bindings are in the wrong order (this was our bug — garbage like <code>-,unlintzegesenma</code>)</li>
  <li>Wrong uniform values for a shader (K_groups, N, etc.)</li>
</ul>

<h3>WebGPU validation error about aliasing</h3>
<ul>
  <li>Same buffer bound as <code>read_write</code> and <code>read</code> in one dispatch</li>
  <li>Fix: use ping-pong buffers. Never bind <code>B.residual</code> as both <code>@1</code> and <code>@4</code> to <code>add_norm</code></li>
</ul>

<h3>Model not loading (<code>Weight not found</code>)</h3>
<ul>
  <li>Log all available keys: the weight loader prints them to console on load</li>
  <li>MLC names differ from HuggingFace standard names (<code>transformer.h.N.mixer.*</code> not <code>model.layers.N.self_attn.*</code>)</li>
</ul>

<!-- BUGS FIXED -->
<h2 id="bugs-fixed">Bugs we fixed (and how)</h2>

<div class="table-scroll"><table>
  <thead><tr><th>Bug</th><th>Symptom</th><th>Fix</th></tr></thead>
  <tbody>
    <tr>
      <td>Wrong MLC param names</td>
      <td><code>Weight not found</code> error on load</td>
      <td>Logged all 325 param names from console, updated candidates to <code>transformer.h.*</code> prefix</td>
    </tr>
    <tr>
      <td>Buffer aliasing in add_norm</td>
      <td><code>&lt;unk&gt;</code> × 500 at 314 tok/s + WebGPU validation error</td>
      <td>Added <code>B.residual2</code> (pong buffer), ping-pong with JS variable swap</td>
    </tr>
    <tr>
      <td>Wrong rope binding order</td>
      <td>Garbage: <code>-,unlintzegesenma\dOCĆalloqueIAL</code> repeated</td>
      <td>Read rope.wgsl — bindings are <code>@0=q_out @1=k_out @2=v_out @3=qkv @4=posMap</code></td>
    </tr>
    <tr>
      <td>Mixed ndarray-cache.json version</td>
      <td><code>&lt;unk&gt;</code> after downloading index locally but using old cached shards</td>
      <td>Always fetch index and shards from the same source atomically</td>
    </tr>
  </tbody>
</table></div>

<!-- VS WEBLLM -->
<h2 id="vs-webllm">vs WebLLM</h2>

<p>Head-to-head on Phi-3-mini-4k-instruct Q4F16_1, same weights, same session, same browser (Chrome 150 with WebGPU), Apple M2 Max, WebLLM 0.2.80 (the Qwen pairs below ran against 0.2.84) — <code>npm run bench</code>, 128-token target × 5 runs, median, 2026-07-30 corrected protocol (latest run recorded in <code>bench/results.json</code>). Every run pays a full prefill on both sides, and both metrics are reported: <em>total</em> is wall-clock throughput including prefill, <em>decode</em> excludes it.</p>

<div class="table-scroll"><table>
  <thead><tr><th></th><th>Zero-TVM</th><th>WebLLM</th></tr></thead>
  <tbody>
    <tr><td>Total throughput (prefill + decode)</td><td><!--bench:zt-->69.55<!--/bench:zt--> tok/s</td><td><!--bench:webllm-->60.0<!--/bench:webllm--> tok/s</td></tr>
    <tr><td>Decode only</td><td>83.10 tok/s</td><td>63.23 tok/s (self-reported)</td></tr>
    <tr><td>Time to first token (~35-token prompt)</td><td>291 ms</td><td><strong>~150 ms</strong> (implied from 251 tok/s prefill) — WebLLM ahead</td></tr>
    <tr><td>Gap</td><td colspan="2"><!--bench:gap-->-16<!--/bench:gap-->% on total relative to WebLLM (negative = Zero-TVM ahead), i.e. +16.0% total and +31.4% decode-only. Same-session pair; cross-session absolute tok/s drifts and the old "−28…−31% stable band" was retired on 2026-07-30 (see BENCH.md).</td></tr>
    <tr><td>Dispatches / token</td><td>260 default (228 with <code>?splitk=0</code>)</td><td>342</td></tr>
    <tr><td>Distinct shaders</td><td>10 hand-written kernel roles</td><td>~11 TVM-generated on the decode path (85 captured across a session)</td></tr>
    <tr><td>Shipped JS bundle</td><td>~460 kB / ~126 kB gz (<code>zero-tvm.html</code>)</td><td>~6.0 MB / ~2.2 MB gz (@mlc-ai/web-llm 0.2.84)</td></tr>
    <tr><td>Bandwidth utilization (total / decode)</td><td>~36% / ~44% of the 191 tok/s ceiling</td><td>~31% / ~33% of the same ceiling</td></tr>
    <tr><td>Paged attention</td><td>✓ Hand-written</td><td>✓ TVM compiled</td></tr>
    <tr><td>Readable kernels?</td><td>✓ Yes — every .wgsl file is in the repo</td><td>✗ No — emitted by the TVM compiler</td></tr>
  </tbody>
</table></div>

<h3>The corrected protocol (2026-07-30) — and what it invalidated</h3>

<div class="callout warning">
  <div class="callout-title">Our half of the A/B had stopped paying prefill</div>
  <p><code>bench()</code> in <code>src/zero-tvm/bench-console.ts</code> looped its runs against the same prompt and never called <code>engine.resetKVTracking()</code> — while <code>benchPrefill()</code>, <code>specSim()</code> and <code>validate.ts</code> all did. That was harmless until cross-turn prefix reuse shipped on 2026-07-29 (PR #24). After it, runs 2..N of every bench found the whole prompt already absorbed and prefilled exactly <strong>one</strong> token, while the WebLLM half — a fresh chat completion per run — kept paying a full prefill inside its wall clock. The two halves were measuring different work.</p>
  <p><strong>Withdrawn as a result:</strong> the Qwen3-4B "75.74 / 43.75, +73.1%" and Qwen3.5-4B "65.67 / 34.04, +92.9%" pairs, both 2026-07-29. Everything earlier predates prefix reuse and was like-for-like. <strong>Fixed:</strong> <code>bench()</code> resets before every run, both halves split TTFT from decode, and WebLLM's own per-run decode/prefill rates are captured instead of logged once and discarded. Prior published numbers stay in BENCH.md as dated history with the defect explained in place — nothing was silently overwritten.</p>
</div>

<h3>The advantage grows with architecture recency</h3>

<p>Monotonic on both metrics across the three models then shipped, all measured 2026-07-30 on the same machine (Qwen3.6-35B-A3B came later and has no WebLLM baseline to pair against — see BENCH.md):</p>

<div class="table-scroll"><table>
  <thead><tr><th>Model</th><th>Architecture released</th><th>Δ total</th><th>Δ decode</th></tr></thead>
  <tbody>
    <tr><td>Phi-3-mini</td><td>2024</td><td>+16.0%</td><td>+31.4%</td></tr>
    <tr><td>Qwen3-4B</td><td>2025</td><td>+31.7%</td><td>+58.0%</td></tr>
    <tr><td>Qwen3.5-4B</td><td>2026</td><td>+100.5%</td><td>+113.6%</td></tr>
  </tbody>
</table></div>

<p>The reading that fits: compiler stacks have had less time to tune newer architectures, so there is more headroom for a hand-written kernel set to take. That is an <strong>observation across three points</strong>, not a proven law — one machine, one browser, one day, no mechanism isolated, no control for how differently each model stresses the two engines — and the baseline is not constant across the rows (WebLLM 0.2.80 for Phi-3, 0.2.84 for both Qwen rows). Worth testing on a fourth model.</p>

<h3>Where we lose: time to first token on short prompts</h3>

<div class="callout warning">
  <div class="callout-title">WebLLM reaches the first token sooner than we do</div>
  <p>WebLLM's self-reported prefill runs at 251 tok/s on Phi-3 and 263–271 tok/s on Qwen3-4B. Against the ~35-token bench prompt that implies a TTFT of roughly <strong>150 ms</strong> on Phi-3, where we measure <strong>291 ms</strong>; on Qwen3-4B our <strong>453 ms</strong> is worse still. Only Qwen3.5-4B is a wash (our 171 ms vs an implied ~0.2 s). Stated plainly: <strong>we win sustained decode decisively and lose the first-token sprint on short inputs.</strong></p>
  <p>It is specifically a <em>short</em>-prompt weakness. Chunked prefill measures 202 tok/s on an 816-token prompt (2026-07-29), and cross-turn prefix reuse removes prefill entirely on follow-up turns. What is not competitive is the fixed cost of a short, cold prefill with nothing cached to reuse. It is the top open item on BENCH.md's levers list.</p>
</div>

<h3>How the result flipped (22% behind &rarr; +16% total / +31% decode ahead)</h3>
<p>An earlier head-to-head (M2 Pro, 2026-06) read 42.14 vs ~51.5 tok/s — Zero-TVM 22% <em>behind</em>. Both the hardware and the engine changed since, so the two Zero-TVM numbers are not a pure optimization delta; the same-session, same-machine WebLLM number is the valid comparator for the current Zero-TVM median (both in the table above). What changed in the engine:</p>
<ul>
  <li><strong>Correctness fixes</strong>. f32 accumulation in the fused FFN, a workgroup-barrier fix in attention, and a decode off-by-one fix — the old number understates even the old hardware.</li>
  <li><strong>vec4 loads, now default</strong>. Re-declaring weight + activation buffers as <code>array&lt;vec4&lt;u32&gt;&gt;</code> in the int4 matmuls and <code>qkv_fused</code> measured +7.1% on M2 Max; opt out with <code>?vec4=0</code> / <code>?vec4qkv=0</code>.</li>
  <li><strong>Tiled + subgroup kernels</strong>. The 4-row tiled subgroup FFN and the <code>_sg</code> matmul/attention/argmax variants are the default path on Apple sg32 hardware.</li>
</ul>
<p>Still default since 2026-07-27 (`?splitk=0` to disable): split-K attention (<code>?splitk=N</code>, ~+3% at short context, needs a long-context A/B); the best measured opt-in config of that era (<code>?vec4=1&amp;vec4qkv=1&amp;splitk=8</code>, 68.36 tok/s on 2026-07-25) is now simply the default path. Falsified and not shipped: FFN prologue fusion (<code>?fuseprologue=1</code>, −13.7% on M2 Max). Dispatch overhead was never the gap — Zero-TVM already submitted fewer dispatches per token (260 vs 342) when it was slower. Full A/B tables in BENCH.md.</p>

<h3>The memory-bandwidth ceiling</h3>
<p>Phi-3-mini Q4F16_1 touches ~2.09 GB of weights per decode token (the weight-shape table above, summed: 32 layers plus the LM head; the embedding contributes one row). On M2 Max's 400 GB/s memory bus that's ~5.2 ms/token, or ~191 tok/s theoretical max. Zero-TVM's measured decode-only median sits at roughly 44% of that ceiling (36% on total, which pays prefill too); WebLLM's at roughly 33% decode-only / 31% total (exact medians in the table above). Neither engine can exceed the ceiling without changing weight layout or quantization (int8 KV shaves a bit, which is why it's exposed behind a flag).</p>

</main>
</div>

<script>
// Highlight active sidebar link on scroll
const links = document.querySelectorAll('.sidebar-link[href^="#"]')
const observer = new IntersectionObserver((entries) => {
  for (const entry of entries) {
    if (entry.isIntersecting) {
      const id = entry.target.id
      links.forEach(l => {
        l.classList.toggle('active', l.getAttribute('href') === '#' + id)
      })
    }
  }
}, { rootMargin: '-60px 0px -60% 0px' })

document.querySelectorAll('[id]').forEach(el => observer.observe(el))
</script>

</body>
</html>