zerotvm.com β pick a model, it runs in your tab.
+
+
+
diff --git a/README.md b/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..54ec500cb2a762420ae018dd09194d107ef4f5e2
--- /dev/null
+++ b/README.md
@@ -0,0 +1,120 @@
+---
+title: Zero-TVM
+emoji: π§
+colorFrom: indigo
+colorTo: purple
+sdk: static
+pinned: true
+license: mit
+thumbnail: https://huggingface.co/spaces/abgunaydin/zero-tvm/resolve/main/og.png
+short_description: Phi-3 to a 35B MoE in the browser, on hand-written WGSL
+tags:
+ - webgpu
+ - llm
+ - inference
+ - on-device
+ - wgsl
+models:
+ - microsoft/Phi-3-mini-4k-instruct
+ - Qwen/Qwen3-4B
+ - Qwen/Qwen3.5-4B
+ - meta-llama/Llama-3.2-1B-Instruct
+ - mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC
+ - mlc-ai/Qwen3-4B-q4f16_1-MLC
+ - mlc-ai/Qwen3.5-4B-q4f16_1-MLC
+ - mlx-community/Qwen3-4B-4bit
+ - mlx-community/Qwen3-30B-A3B-4bit
+ - mlx-community/Llama-3.2-1B-Instruct-4bit
+ - lmstudio-community/Qwen3.6-35B-A3B-MLX-4bit
+ - abgunaydin/Qwen3.6-35B-A3B-MLX-q3exp
+---
+
+# Zero-TVM
+
+Models from a 3.8B dense to a 35B sparse MoE, running in the browser on hand-written WGSL. No TVM, no ONNX, no WASM runtime.
+
+The standard way to run a modern LLM in a browser is [WebLLM / MLC-LLM](https://webllm.mlc.ai/), which ships an Apache-TVM compiler pipeline that autotunes its WGSL β 85 shaders captured across a session, around 11 of them on the decode path. This Space replaces that entire stack with **10 kernel roles** of hand-written WGSL β and runs models WebLLM ships (same weights, measured faster) as well as one it does not: Qwen3.6-35B-A3B, a 256-expert sparse MoE.
+
+The whole forward pass β 32 transformer layers, paged KV cache, int4-dequant matmul, RoPE, fused FFN, RMSNorm, paged attention, argmax sampling β is readable end-to-end in a single sitting. That is the point.
+
+## Headline result
+
+Measured on an Apple M2 Max (Chrome 150, WebLLM v0.2.80, identical Phi-3-mini-q4f16_1 weights, same-session head-to-head, 2026-07-30). Two numbers, always both: **total** is wall-clock throughput with prefill included (both engines doing identical work), **decode** excludes prefill.
+
+| | WebLLM (TVM) | Zero-TVM |
+|---|---|---|
+| WGSL kernels (decode path) | **~11 TVM-generated** (85 captured across a session) | **10 hand-written roles** |
+| Dispatches per decode step | **342** | **260** default (228 with `?splitk=0`) |
+| Total throughput (prefill + decode) | **59.95 tok/s** | **69.55 tok/s** β **+16.0%** |
+| Decode only | **63.23 tok/s** (self-reported) | **83.10 tok/s** β **+31.4%** |
+| Time to first token (~35-tok prompt) | **~150 ms** (implied) | **291 ms** β *WebLLM wins this one* |
+| JS bundle (excl. weights) | **~6.0 MB** / ~2.2 MB gz | **~460 kB** / ~126 kB gz |
+
+**The hand-written kernels don't just keep up with the autotuning compiler β on this hardware they beat it on throughput.** For decoder-only LLMs of this shape, most of the compiler's complexity budget isn't buying much β the expensive parts are matmul, attention, and int4 dequant. Everything else is plumbing, and ten hand-written roles cover it.
+
+**The gap grows with how recent the architecture is** β +16% / +31% (total / decode) on 2024's Phi-3, +32% / +58% on 2025's Qwen3-4B, +100% / +114% on 2026's Qwen3.5-4B. Consistent with compiler stacks having had less time to tune newer architectures; an observation across three points on one machine, not a proven law.
+
+**Honest negative, published not buried:** WebLLM reaches the first token faster than we do on short prompts (~150 ms implied vs our 291 ms on Phi-3; 453 ms on Qwen3-4B). We win sustained throughput and lose the first-token sprint on short inputs β our chunked prefill is strong on long ones (202 tok/s at 816 tokens) and prefix reuse makes follow-up turns free.
+
+Exact medians, full methodology, and the optimization experiments that were measured and *dropped* live in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md). (An earlier measurement on an M2 Pro with an older, buggier engine read 22% *behind* β that history is preserved there too, not retconned. So are two 2026-07-29 Qwen pairs that a bench-harness defect made non-comparable; they are marked withdrawn with the reason, not deleted.)
+
+## Run it
+
+The Space opens on a character-select entrance: pick a model and it loads and chats in place. [`zero-tvm.html`](./zero-tvm.html) is the same chat as a direct link, if you want to deep-link a specific model.
+
+Picking a character starts the download immediately β the sheet shows the weight size before you click, and there is no second confirmation. It streams ~2 GB of Phi-3-mini-q4f16_1 weights from the Hugging Face mirror at [`mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC) into OPFS (Origin Private File System).
+
+Subsequent loads are instant β the weights stay in OPFS, so returning to a model you have already downloaded starts it straight away.
+
+**Requirements**: Chrome / Edge with WebGPU enabled and the `shader-f16` feature available (default on macOS Apple-Silicon, enabled on most modern Windows / Linux GPUs).
+
+## Models
+
+All three pairs re-measured 2026-07-30 under the corrected protocol β same session, identical local weight bytes, both engines paying a full prefill on every run, total and decode reported side by side.
+
+- **Phi-3-mini-4k-instruct (3.8B, q4f16_1)** β the default; every headline number above is Phi-3, no URL flag needed. Zero-TVM **69.55 tok/s total** (TTFT 291 ms, decode 83.10) vs WebLLM 0.2.80's **59.95 tok/s total** (self-reported decode 63.23) β **+16.0% total, +31.4% decode**.
+- **Qwen3-4B (q4f16_1)** β append `?model=qwen3` to `zero-tvm.html` or `validate.html`; weights stream from [`mlc-ai/Qwen3-4B-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Qwen3-4B-q4f16_1-MLC) (~2.3 GB). A port on the spec-parameterized engine: GQA 32/8, per-head QK-norm, byte-level BPE, tied lm_head, with the tuning round's fused post-matmul chain (`qk_norm_rope_append`, 8 dispatches/layer) and K%512 wide loads. Zero-TVM **59.85 tok/s total** (TTFT 453 ms, decode 75.49) vs WebLLM 0.2.84's prebuilt Qwen3-4B at **45.46 tok/s total** (self-reported decode 47.77) β **+31.7% total, +58.0% decode**. This is the model where WebLLM most clearly beats us on first-token latency (~150 ms implied vs our 453 ms). The previously published "75.7 vs 43.8, +73%" pair is **withdrawn** β a bench-harness defect had stopped our half paying prefill; and the 2026-07-28 pair (25.43 vs 14.15) was a degraded session that did not reproduce. Both are kept as dated history with the reason in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md).
+- **Qwen3.5-4B (q4f16_1)** β append `?model=qwen35` to `zero-tvm.html` or `validate.html`; weights stream from [`mlc-ai/Qwen3.5-4B-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Qwen3.5-4B-q4f16_1-MLC) (~2.6 GB). The first *hybrid* on the engine: 24 gated-DeltaNet (linear-attention) layers + 8 gated-attention layers (GQA 16/4, head_dim 256, partial RoPE, sigmoid attention gate) β to our knowledge the first hand-written-kernel int4 gated-DeltaNet hybrid in a browser. Zero-TVM **65.28 tok/s total** (TTFT 171 ms, decode 73.30) vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at **32.56 tok/s total** (self-reported decode 34.32) β **+100.5% total, +113.6% decode**, and the one model where first-token latency is a wash rather than a loss. The GDN decode kernels are still scalar (non-subgroup), so the decode number is a floor; prefill runs in chunks of β€64 (202 tok/s on an 816-token prompt) and every model reuses its cross-turn prefix (turn-3 first token 14.3 s β 0.19 s, logits verified bit-identical to a fresh prefill). The previously published "65.7 vs 34.0, +93%" cross-check is **withdrawn** (same harness defect); the earlier 53.07/32.36 (+64%) and 47.99/31.99 (+50%) pairs were like-for-like but are superseded. One machine, one pair each; caveats in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md).
+
+## Pages in this Space
+
+- [`index.html`](./index.html) β the entrance: pick a model, chat in place (start here)
+- [`zero-tvm.html`](./zero-tvm.html) β the chat surface as a direct link
+- [`validate.html`](./validate.html) β multi-prompt smoke test
+- [`docs.html`](./docs.html) β annotated reference, including the kernel walkthrough
+
+Six pages were removed on 2026-08-14 β `architecture`, `demo`, `compiler-chat`,
+`dump`, `shaders`, `webllm-bench`. Three of them started multi-gigabyte
+downloads on page load. `docs.html` carries what the first two explained.
+
+## URL flags
+
+`zero-tvm.html` accepts query flags for A/B-ing shader variants without rebuilding:
+
+- `?sg=0` β disable all subgroup shaders (argmax / attention / QKV matmul)
+- `?sgqkv=0` / `?sgattn=0` / `?sgargmax=0` β disable one at a time
+- `?qkvtile=1` / `?qkvtile2=1` β opt into tiled QKV variants
+- `?ffnsg=1` β opt into the tiled-subgroup fused FFN
+- `?kv8=1` β opt into the int8 KV cache path
+
+## How it relates to a published paper
+
+Zero-TVM is the LLM-decoding application of a broader thesis: **per-dispatch overhead is the dominant tax on browser GPU compute, and most workloads collapse from "needs a server" to "runs in a browser tab" once you fuse the loop.** Two preprints document the underlying technique:
+
+- [Single-Kernel Fusion for Sequential Fitness Evaluation via WebGPU Compute Shaders](https://doi.org/10.5281/zenodo.19342888) (the foundational result)
+- [Single-Kernel Fusion for Autoregressive Transformer Decoding via WebGPU Compute Shaders](https://doi.org/10.5281/zenodo.19344277) (the LLM application β the work this Space embodies)
+
+## Source / canonical site
+
+- **GitHub**: [github.com/abgnydn/zero-tvm](https://github.com/abgnydn/zero-tvm)
+- **Canonical site**: [zerotvm.com](https://zerotvm.com)
+- **Research umbrella**: [kernelfusion.dev](https://kernelfusion.dev)
+- **Companion benchmarks**: [gpubench.dev](https://gpubench.dev)
+
+## License
+
+MIT. Reference comparison code (WebLLM compiler-chat path) belongs to its respective authors and is governed by their licenses.
+
+## Author
+
+Ahmet BarΔ±Ε GΓΌnaydΔ±n Β· [barisgunaydin.com](https://barisgunaydin.com) Β· senior full-stack consultant by day, independent researcher by night.
diff --git a/_headers b/_headers
new file mode 100644
index 0000000000000000000000000000000000000000..f223119b309fe0abf4ca23aa85420c40e2d93ee2
--- /dev/null
+++ b/_headers
@@ -0,0 +1,8 @@
+# Cloudflare Pages headers (migrated from the old vercel.json).
+# Cross-origin isolation is required for SharedArrayBuffer (WebGPU itself does not need it).
+/*
+ Cross-Origin-Opener-Policy: same-origin
+ Cross-Origin-Embedder-Policy: credentialless
+
+/assets/*
+ Cache-Control: public, max-age=31536000, immutable
diff --git a/_redirects b/_redirects
new file mode 100644
index 0000000000000000000000000000000000000000..f016ca946e9d6a435295d96c610c18f3da64304c
--- /dev/null
+++ b/_redirects
@@ -0,0 +1,16 @@
+# Six pages deleted in the 2026-08-14 redesign, five of them in the sitemap
+# Google is crawling at priority 0.6-0.9. Without these, every bookmark, README
+# link and search result for them lands on a bare 404 and the accumulated link
+# equity is dropped instead of passed to the successor page.
+/demo / 301
+/demo.html / 301
+/architecture /docs 301
+/architecture.html /docs 301
+/shaders /docs#shaders 301
+/shaders.html /docs#shaders 301
+/dump /docs 301
+/dump.html /docs 301
+/compiler-chat /zero-tvm 301
+/compiler-chat.html /zero-tvm 301
+/webllm-bench /docs#vs-webllm 301
+/webllm-bench.html /docs#vs-webllm 301
diff --git a/agent-host.html b/agent-host.html
new file mode 100644
index 0000000000000000000000000000000000000000..0441f2b1d7e6744ee72b2b75001c5045275c1e39
--- /dev/null
+++ b/agent-host.html
@@ -0,0 +1,78 @@
+
+
+
+
+
+
+
+
+
+zero-tvm Β· agent host
+
+
+
+
+
+
+
+
zero-tvm Β· agent host
+
This tab runs the model. scripts/agent-server.mjs gives it an OpenAI-shaped address on :8017.
+ Alternative, and the one to prefer for real use: npm run station runs the engine in-process on
+ dawn.node (no browser tab, no throttling) and takes the same :8017. They cannot both hold the port β run one.
+
+
+
bootingβ¦
+
+
+
+
+
Point a client at http://127.0.0.1:8017/v1
+
+ Cline β API Provider βOpenAI Compatibleβ, Base URL above, any model id.
+ pi β ~/.pi/agent/models.json, api: "openai-completions", that baseUrl.
+ curl β curl 127.0.0.1:8017/v1/chat/completions -d '{"messages":[{"role":"user","content":"hi"}]}' -H 'content-type: application/json'
+
+
+
+
+
Keep this tab visible.
+
Chrome throttles background tabs hard β share.ts measured ~23 tok/s
+ backgrounded against ~65 focused. An agent run is long and unattended, which is exactly when a tab
+ stops being focused.
+
+
+
+
+
+
+
diff --git a/assets/agent-host-eDUnNJBhJx.js b/assets/agent-host-eDUnNJBhJx.js
new file mode 100644
index 0000000000000000000000000000000000000000..04d09595af98c8669f33a0c29b014da57b469e39
--- /dev/null
+++ b/assets/agent-host-eDUnNJBhJx.js
@@ -0,0 +1,84 @@
+const __vite__mapDeps=(i,m=__vite__mapDeps,d=(m.f||(m.f=["assets/cache-probe-BYX_n9OPRB.js","assets/weight-loader-ByOLMd9f5C.js","assets/model-registry-BoK-ObgACo.js"])))=>i.map(i=>d[i]);
+var te=Object.defineProperty;var ne=(e,t,n)=>t in e?te(e,t,{enumerable:!0,configurable:!0,writable:!0,value:n}):e[t]=n;var P=(e,t,n)=>ne(e,typeof t!="symbol"?t+"":t,n);import{m as re}from"./model-registry-BoK-ObgACo.js";import{_ as oe}from"./preload-helper-ckwbz45pO3.js";import{bootEngine as ae}from"./loading-ui-CjCwSRuXvh.js";import{buildDecodeEngine as se,allocKVPages as ie}from"./engine-core-CLR3cxsqf4.js";import{parseVariantFlags as le}from"./variants-De-VVc9BRz.js";import{specFromSearch as ce,buildChatPromptFor as ue}from"./model-select-DhhV9mCi0T.js";import"./weight-loader-ByOLMd9f5C.js";const M=0xffffffffffffffffn,fe=0x243f6a8885a308d3n,me=0x13198a2e03707344n;function he(e){return e=(e^e>>30n)*0xbf58476d1ce4e5b9n&M,e=(e^e>>27n)*0x94d049bb133111ebn&M,(e^e>>31n)&M}function X(e,t){let n=e;for(let o=0;o>>0)+0x9e3779b97f4a7c15n&M);return n}function pe(e){const t=X(fe,e),n=X(me,e);return t.toString(16).padStart(16,"0")+n.toString(16).padStart(16,"0")}const de=new Set(["splitK"]),ge="zero-tvm/kv-prefix/1";function ye(e){const t=e.variants,n=Object.keys(t).filter(o=>!de.has(o)).sort().map(o=>` ${o}=${String(t[o])}`);return[ge,`spec=${e.specId}`,`weights=${e.weightRevision}`,`fused=${e.fused}`,`int8kv=${e.int8KV}`,"layers=all",`prefill=${e.prefillPath}`,`adapter=${e.adapter.vendor}|${e.adapter.architecture}`,"variants:",...n].join(`
+`)}function we(e){const t=ye(e),n=new Array(t.length);for(let o=0;ol.byteLength),savedAt:Date.now()},a=await(await o.getFileHandle("entry.bin",{create:!0})).createWritable();for(const l of n.layers)await a.write(l);for(const l of n.scales)await a.write(l);for(const l of n.gdn)await a.write(l);await a.close();const i=await(await o.getFileHandle("meta.json",{create:!0})).createWritable();return await i.write(JSON.stringify(r)),await i.close(),{tokens:n.tokens}}async function xe(e,t,n){const o=await z(Y(t),!1);if(!o)return{restored:0,reason:"no entry for this fingerprint"};let r,a;try{r=JSON.parse(await(await(await o.getFileHandle("meta.json")).getFile()).text()),a=await(await(await o.getFileHandle("entry.bin")).getFile()).arrayBuffer()}catch{return{restored:0,reason:"no entry for this fingerprint"}}if(r.format!==F)return{restored:0,reason:`entry format ${r.format} != ${F}`};if(r.tokens>=n.length)return{restored:0,reason:"prompt does not extend the entry"};for(let u=0;uu+g,0);if(a.byteLength!==i)return{restored:0,reason:"entry truncated"};const f=[];let s=0;for(let u=0;u0)for(let u=0;u XML tags:
+`,ve=`
+
+
+For each function call, return a json object with function name and arguments within XML tags:
+
+{"name": , "arguments": }
+`,_e=`# Tools
+
+You have access to the following functions:
+
+`,Oe=`
+
+
+If you choose to call a function ONLY reply in the following format with NO suffix:
+
+
+
+
+value_1
+
+
+This is the value for the second parameter
+that can span
+multiple lines
+
+
+
+
+
+Reminder:
+- Function calls MUST follow the specified format: an inner block must be nested within XML tags
+- Required parameters MUST be specified
+- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after
+- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
+`,Se=`Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.
+
+Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.Do not use variables.
+
+`;function Le(e,t,n){var s;if(n.length===0)return t.slice();const o=n.map(l=>`
+${_(l)}`).join("");if(e==="llama3"){const l=t.slice(),h=l.findIndex(u=>u.role==="user");if(h<0)throw new Error("Cannot put tools in the first user message when there is no user message");const $=n.map(u=>`${JSON.stringify(u,null,4)}
+
+`).join("");return l[h]={role:"user",content:Se+$+l[h].content.trim()},l}const r=e==="chatml-xml"?_e+o+Oe:ke+o+ve,a=t.slice(),i=((s=a[0])==null?void 0:s.role)==="system"?a[0].content:null;let f;if(e==="chatml-xml"){const l=(i??"").trim();f=l?`${r}
+
+${l}`:r}else f=i?`${i}
+
+${r}`:r;return i===null?a.unshift({role:"system",content:f}):a[0]={role:"system",content:f},a}function Ee(e,t,n){if(n.length===0)return t;if(e==="llama3"){if(n.length>1)throw new Error("llama3 renders a single tool call per assistant message");return`{"name": "${n[0].name}", "parameters": ${_(n[0].arguments)}}`}if(e==="chatml-xml"){let r=t.trim();return n.forEach((a,i)=>{r+=i===0?r?`
+
+`:"":`
+`,r+=`
+
+`;for(const[f,s]of Object.entries(a.arguments))r+=`
+${typeof s=="string"?s:_(s)}
+
+`;r+=`
+`}),r}let o=t;return n.forEach((r,a)=>{(a>0||t)&&(o+=`
+`),o+=`
+{"name": "${r.name}", "arguments": ${_(r.arguments)}}
+`}),o}const B="",K="";function G(e,t){const n=e.indexOf("{",t);if(n<0)return null;let o=0,r=!1,a=!1;for(let i=n;i0;r--)if(e.endsWith(n.slice(0,r)))return r;return 0}function Q(e,t){if(typeof e!="object"||e===null||Array.isArray(e))return{name:"",arguments:{},raw:t,error:"tool call is not a JSON object"};const n=e;if(typeof n.name!="string"||n.name==="")return{name:"",arguments:{},raw:t,error:'tool call has no "name"'};let o=n.arguments??n.parameters;if(typeof o=="string")try{o=JSON.parse(o)}catch{}return o==null?{name:n.name,arguments:{},raw:t}:typeof o!="object"||Array.isArray(o)?{name:n.name,arguments:{},raw:t,error:"tool call arguments are not an object"}:{name:n.name,arguments:o,raw:t}}function Ce(e){const t=G(e,0);if(!t||t.end<0)return{name:"",arguments:{},raw:e,error:"no JSON object in block"};const n=e.slice(t.start,t.end);let o;try{o=JSON.parse(n)}catch(r){return{name:"",arguments:{},raw:e,error:`invalid JSON in block: ${r.message}`}}return Q(o,e)}function Ae(e){const t=e.indexOf("",t);if(t<0||n<0)return{name:"",arguments:{},raw:e,error:"no in block"};const o=e.slice(t+10,n),r=e.indexOf("",n);if(r<0)return{name:o,arguments:{},raw:e,error:"unterminated block"};const a={},i=e.slice(n+1,r);let f=0;for(;;){const s=i.indexOf("",s);if(l<0)return{name:o,arguments:a,raw:e,error:"unterminated tag"};let h=-1;for(let g=i.indexOf("",l);g>=0;g=i.indexOf("",g+1)){const C=i.slice(g+12).trim();if(C===""||C.startsWith(" value"};const $=i.slice(s+11,l);let u=i.slice(l+1,h);u.startsWith(`
+`)&&(u=u.slice(1)),u.endsWith(`
+`)&&(u=u.slice(0,-1)),a[$]=Ne(u),f=h+12}return{name:o,arguments:a,raw:e}}function Ne(e){try{const t=JSON.parse(e);return typeof t=="string"?e:t}catch{return e}}function Re(e,t){return e==="llama3"?Pe(t):Me(e,t)}function Me(e,t){const n=[];let o="",r=!1,a=0;for(;a=2;break}o+=t.slice(a,i);const f=i+B.length,s=t.indexOf(K,f);if(s<0){r=!0;break}const l=t.slice(f,s);n.push(e==="chatml-xml"?Ae(l):Ce(l)),a=s+K.length}return{text:o,calls:n,partial:r}}function Pe(e){const t=[];let n="",o=!1,r=0;for(;r")&&(i=i.slice(0,-14)),a.end<0){n+=i,o=!0;break}const f=e.slice(a.start,a.end);let s=null;try{const l=Q(JSON.parse(f),f);l.error||(s=l)}catch{}n+=s?i:e.slice(r,a.end),s&&t.push(s),r=a.end}return{text:n,calls:t,partial:o}}const R=new URL(location.href).searchParams.get("server")??"http://127.0.0.1:8017",J=e=>document.getElementById(e),T=(e,t="")=>{const n=J("status");n.textContent=e,n.className=t},w=e=>{const t=J("log");t.textContent=`${new Date().toLocaleTimeString()} ${e}
+${t.textContent}`.split(`
+`).slice(0,200).join(`
+`)},m=ce(location.search),U=re(m);J("model").textContent=`${U.name} β ${U.params}`;const Be=new Set(["qwen36","qwen3-8-27b"]);function Fe(e){return e==="llama3"?"llama3":[...Be].some(t=>m.id.startsWith(t))?"chatml-xml":"chatml-json"}async function Je(){const{isModelCached:e}=await oe(async()=>{const{isModelCached:n}=await import("./cache-probe-BYX_n9OPRB.js");return{isModelCached:n}},__vite__mapDeps([0,1,2]));if(await e(m))return;const t=document.createElement("dialog");t.id="agent-gate",t.style.cssText="width:calc(100vw - 2rem);max-width:32rem;background:#0f1216;color:#d6dbe1;border:1px solid #21262d;border-radius:10px;padding:1.5rem 1.6rem;font:inherit",t.innerHTML=`
+
Run ${m.id} in this tab?
+
+ This is a developer surface. It downloads the model
+ and then talks to an agent-server on 127.0.0.1:8017 β
+ start it with npm run agent from the repo. Without that
+ server running, this page has nothing to serve.
+
`,document.body.appendChild(t),t.showModal(),await new Promise(n=>{var o;(o=t.querySelector("#agent-gate-go"))==null||o.addEventListener("click",()=>{t.close(),t.remove(),n()},{once:!0})})}async function Ie(){let e=null;if(!("gpu"in navigator)){T("cannot boot: this browser has no WebGPU β Chrome and Edge ship it","err");return}await Je();const t=await ae({spec:m,optionalFeatures:["subgroups","chromium-experimental-subgroup-matrix"],probeSubgroups:!0,buildEngine:({device:c,weights:d,sgSizeOk:O,spec:S})=>(e=le(location.search,{hasSubgroupsFeature:c.features.has("subgroups"),sgSizeOk:O}),se(c,d,ie(c,S),{spec:S,variants:e}))});if(!t.ok)throw T(`cannot boot: ${t.reason}`,"err"),new Error(t.reason);const{engine:n,tokenizer:o}=t,r=Fe(m.chatTemplateId),a=new URL(location.href).searchParams.get("pool")!=="0",i=()=>({spec:m,weightRevision:m.hfRepo,variants:e,fused:!1,int8KV:!1,prefillPath:!m.moe&&e.subgroups?"chunked":"per-token",adapter:{vendor:"browser",architecture:"webgpu"}});let f=!1,s=null,l=null;const h=()=>{a&&(s!==null&&clearTimeout(s),s=self.setTimeout(()=>{s=null,!g&&(l=(async()=>{try{const c=await $e(n,i());c.tokens&&w(`pool: saved ${c.tokens} tokens`)}catch(c){w(`pool: save failed (${c.message})`)}})().finally(()=>{l=null}))},1500))};T(`ready β ${m.maxContext.toLocaleString()} token context, ${r}`,"ok"),w(`engine up: ${m.id}, dialect ${r}`);class ${constructor(d){P(this,"buf","");P(this,"timer",null);this.id=d}push(d){this.buf+=d,this.timer===null&&(this.timer=self.setTimeout(()=>{this.timer=null,this.flush()},60))}async flush(){if(this.timer!==null&&(clearTimeout(this.timer),this.timer=null),!this.buf)return;const d=this.buf;this.buf="",await u({type:"delta",id:this.id,text:d})}}async function u(c){try{await fetch(`${R}/agent/emit`,{method:"POST",headers:{"content-type":"application/json"},body:JSON.stringify(c)})}catch(d){w(`emit failed: ${d.message}`)}}let g=!1;async function C(c){var S,I;if(g){await u({type:"error",id:c.id,message:"busy: this tab serves one request at a time"});return}g=!0,s!==null&&(clearTimeout(s),s=null),l&&await l;const d=performance.now(),O=new $(c.id);try{let x=c.messages.map(p=>{var E;if(p.role!=="assistant"||!((E=p.toolCalls)!=null&&E.length))return{role:p.role,content:p.content};const k=p.toolCalls.map(v=>{var j,q;let W={};try{W=JSON.parse(((j=v.function)==null?void 0:j.arguments)||"{}")}catch{}return{name:((q=v.function)==null?void 0:q.name)??"",arguments:W}});return{role:p.role,content:Ee(r,p.content??"",k)}});(S=c.tools)!=null&&S.length&&(x=Le(r,x,c.tools));const y=ue(m,x,o);if(a&&!f){f=!0;const p=await xe(n,i(),y);w(p.restored?`pool: restored ${p.restored} tokens`:`pool: miss (${p.reason})`)}if(y.length>=m.maxContext)throw new Error(`prompt is ${y.length} tokens, context is ${m.maxContext}. Lower the client's context window, or use a spec with a bigger one (qwen35 and llama32 are 32,768).`);const D=m.maxContext-y.length,H=Math.max(1,Math.min(c.maxTokens??D,D));n.setSampling(c.temperature!=null&&c.temperature>0?{temperature:c.temperature,topP:c.topP??1}:null);let b="";const A={stop:null},L=await n.generatePipelined(y,H,p=>{var E;const k=o.decode([p]);if(b+=k,O.push(k),(E=c.stop)!=null&&E.length){for(const v of c.stop)if(v&&b.endsWith(v)){A.stop=v;break}}},()=>A.stop!==null);await O.flush(),A.stop&&(b=b.slice(0,b.length-A.stop.length));const V=(I=c.tools)!=null&&I.length?Re(r,b):{text:b,calls:[]},N=V.calls.map((p,k)=>({id:`call_${c.id.slice(0,8)}_${k}`,type:"function",function:{name:p.name,arguments:JSON.stringify(p.arguments??{})}})),ee=(performance.now()-d)/1e3;w(`${c.id.slice(0,8)} ${y.length} in / ${L.length} out ${(L.length/ee).toFixed(1)} tok/s`+(N.length?` ${N.length} tool call(s)`:"")),await u({type:"done",id:c.id,text:V.text,toolCalls:N,finishReason:N.length?"tool_calls":L.length>=H?"length":"stop",usage:{prompt_tokens:y.length,completion_tokens:L.length,total_tokens:y.length+L.length}})}catch(x){await O.flush(),w(`${c.id.slice(0,8)} ERROR ${x.message}`),await u({type:"error",id:c.id,message:x.message})}finally{g=!1,h()}}function Z(){const c=new EventSource(`${R}/agent/jobs?model=${encodeURIComponent(m.id)}`);c.onopen=()=>{T(`hosting ${m.id} β ${R}/v1`,"ok"),w("connected to agent-server")},c.onmessage=d=>{C(JSON.parse(d.data))},c.onerror=()=>{T(`agent-server unreachable at ${R} β retrying`,"err")}}Z()}Ie();
diff --git a/assets/cache-probe-BYX_n9OPRB.js b/assets/cache-probe-BYX_n9OPRB.js
new file mode 100644
index 0000000000000000000000000000000000000000..06af5ff0a2f8e60dffa37f1d73b8d700da238a09
--- /dev/null
+++ b/assets/cache-probe-BYX_n9OPRB.js
@@ -0,0 +1 @@
+import{p as s,a as l,o as f}from"./weight-loader-ByOLMd9f5C.js";import{o as c}from"./model-registry-BoK-ObgACo.js";async function g(t){return t.weightFormat!=="mlx-safetensors",!1}async function y(t){var n;if(typeof navigator>"u"||!((n=navigator.storage)!=null&&n.getDirectory))return!1;try{const o=await(await navigator.storage.getDirectory()).getDirectoryHandle(c(t));if(t.weightFormat==="mlx-safetensors"){const e=s(t),r=e[e.length-1];await o.getFileHandle(l(r.plan,r.layer))}else{const e=await o.getFileHandle(t.manifestName??"ndarray-cache.json"),r=JSON.parse(await(await e.getFile()).text()),i=[...new Set((r.records??[]).map(a=>a.dataPath).filter(a=>!!a))];if(i.length===0)return!1;for(const a of i)await o.getFileHandle(f(a))}return!0}catch{return await g(t)}}export{y as isModelCached};
diff --git a/assets/chat-flow-eTAzKAlIYe.js b/assets/chat-flow-eTAzKAlIYe.js
new file mode 100644
index 0000000000000000000000000000000000000000..6d1b164145c7a450c966b8635c19e8197c62e9a2
--- /dev/null
+++ b/assets/chat-flow-eTAzKAlIYe.js
@@ -0,0 +1,6 @@
+import{buildChatPromptFor as B}from"./model-select-DhhV9mCi0T.js";import{allocKVPagesInt8 as ce,allocKVPages as de,buildDecodeEngine as ue}from"./engine-core-CLR3cxsqf4.js";import{parseVariantFlags as he}from"./variants-De-VVc9BRz.js";import{bootEngine as pe,setBadge as _,log as O}from"./loading-ui-CjCwSRuXvh.js";import{s as fe,q as me,a as Y,w as ge,b as ee,c as te,o as ke}from"./chat-ui-CQAxV1po6S.js";import{m as we}from"./model-registry-BoK-ObgACo.js";function be(t,o,u=3,l=3){let i=o,p=0,g=0,r=0;for(;i=0;a--){if(a+u>e)continue;let m=!0;for(let d=0;d=t.length||d>=t.length||t[m]!==t[d])break;s++}g+=s,i+=1+s}return{totalSteps:p,totalAccepted:g,totalHits:r}}function ye(t,o,u,l=3,i=3){const{totalSteps:p,totalAccepted:g,totalHits:r}=be(o,u,l,i),e=p>0?g/p:0,c=i>0?e/i:0,s=2*(1+e)/(i+1),a=p>0?r/p:0;return{name:t,N:l,K:i,totalSteps:p,totalAccepted:g,meanAcceptedPerStep:e,acceptanceRate:c,theoreticalSpeedup:s,hitRate:a}}const xe="Section %d. The transport of water through a plant begins at the root hairs, where dissolved minerals enter by active transport and water follows by osmosis. Cohesion between water molecules and adhesion to the xylem walls maintain a continuous column that is pulled upward as vapour escapes from the stomata. The rate of that escape depends on humidity, temperature, and wind, so a plant that closes its stomata to conserve water also limits the carbon dioxide available to its chloroplasts. ";function q(t){let o="";for(let u=1;u<=t;u++)o+=xe.replace("%d",String(u));return o}function $e(t,o,u=4){const i=o(q(4)).length,p=o("").length,g=Math.max(1,(i-p)/4);let r=Math.max(0,Math.round((t-p)/g)),e=o(q(r));for(let a=0;a<200&&Math.abs(e.length-t)>u&&!(e.length>t&&r===0||(r+=e.lengtht+u&&s.length>1;a++)s=s.slice(0,-1),c=s.join(" "),e=o(c);for(let a=0;a<400&&e.length{if(t.isBusy())return console.warn("[bench] decode already in flight β skipping"),null;t.setBusy(!0);try{const r=[{role:"system",content:"You are a helpful assistant."},{role:"user",content:"Write a four-sentence explanation of how photosynthesis works."}],e=B(o.spec,r,u);if(p){const f=await o.profileStep(e);return f?(console.log(`[profile] total = ${f.totalMs.toFixed(3)} ms/token across ${f.kernels.reduce((w,T)=>w+T.calls,0)} dispatches`),console.table(f.kernels.map(w=>({kernel:w.label,calls:w.calls,"ms/token":+w.totalMs.toFixed(3),"ms/call":+(w.totalMs/w.calls).toFixed(4),"% total":+w.pctOfTotal.toFixed(1)}))),f):(console.warn("[bench] profile requested but timestamp-query feature unavailable"),null)}const c=[];for(let f=0;f<=i;f++){const w=f===0?"warmup":`run${f}/${i}`;o.resetKVTracking();const T=performance.now();let $=0,P=0;await o.generatePipelined(e,l,()=>{$===0&&(P=performance.now()),$++});const C=performance.now(),A=(C-T)/1e3,R=$/A,D=P?P-T:A*1e3,z=$>1&&P?($-1)/((C-P)/1e3):R;console.log(`[bench] ${w}: ${$} tok / ${A.toFixed(2)}s = ${R.toFixed(2)} tok/s total Β· ttft ${D.toFixed(0)}ms Β· decode ${z.toFixed(2)} tok/s`),f>0&&c.push({tokens:$,seconds:A,tokPerS:R,ttftMs:D,decodeTokPerS:z})}const s=f=>f.slice().sort((w,T)=>w-T)[Math.floor(f.length/2)],a=c.map(f=>f.tokPerS).sort((f,w)=>f-w),m=a[Math.floor(a.length/2)],d=a.reduce((f,w)=>f+w,0)/a.length,b=a[0],S=a[a.length-1],k=s(c.map(f=>f.decodeTokPerS)),x=s(c.map(f=>f.ttftMs)),y={runs:c,median:m,mean:d,min:b,max:S,medianDecode:k,medianTtftMs:x};return console.log(`[bench] summary: median=${m.toFixed(2)} mean=${d.toFixed(2)} min=${b.toFixed(2)} max=${S.toFixed(2)} tok/s Β· decode=${k.toFixed(2)} tok/s Β· ttft=${x.toFixed(0)}ms`),y}finally{t.setBusy(!1),t.onIdle()}},window.benchTtft=async(l=Se,i=8,p=3)=>{if(t.isBusy())return console.warn("[ttft] decode already in flight β skipping"),null;t.setBusy(!0);try{const g=s=>s.slice().sort((a,m)=>a-m)[Math.floor(s.length/2)],r=s=>B(o.spec,[{role:"system",content:"You are a helpful assistant."},{role:"user",content:s}],u),e=[];for(const s of l){const a=$e(s,r),m=r(a.userMessage);if(m.length>o.maxContext){console.warn(`[ttft] target ${s}: ${m.length} tok exceeds maxContext ${o.maxContext} β skipped`);continue}const d=[],b=[];let S=0;o.resetKVTracking(),await o.generatePipelined(m,i,()=>{});for(let k=0;k{y===0&&(f=performance.now()),y++});const w=performance.now(),T=(w-x)/1e3,$=f?f-x:T*1e3,P=y>1&&f?(y-1)/((w-f)/1e3):y/T,C=o.getLastPrefill();C&&(S=C.chunks),d.push($),b.push(P),console.log(`[ttft] prompt=${m.length} tok (target ${s}) run${k+1}: ttft ${$.toFixed(0)}ms Β· ${y} gen tok Β· decode ${P.toFixed(2)} tok/s`+(C?` Β· prefill chunks ${C.chunks} (reused ${C.reused})`:""))}e.push({targetTokens:s,promptTokens:m.length,ttftMsRuns:d,ttftMs:g(d),decodeTokPerS:g(b),chunks:S})}console.log(""),console.log(" prompt tok | TTFT ms | ms/prompt-token | vs shortest | chunks"),console.log(" -----------+----------+------------------+--------------+-------");const c=e[0];for(const s of e)console.log(` ${String(s.promptTokens).padStart(10)} | ${s.ttftMs.toFixed(0).padStart(8)} | ${(s.ttftMs/s.promptTokens).toFixed(3).padStart(16)} | ${(s.ttftMs/c.ttftMs).toFixed(2).padStart(12)}x | ${String(s.chunks).padStart(6)}`);return{mode:"ttft-sweep",engine:"zero-tvm",runsPerPoint:p,genTokensPerPoint:i,points:e}}finally{t.setBusy(!1),t.onIdle()}},window.benchBatched=async(l=4,i=500,p="ffnDown")=>t.isBusy()?(console.warn("[batched-bench] decode in flight β skipping"),null):o.benchBatchedFfnDown(l,i,p),window.specSim=async(l=160,i=3,p=3)=>{if(t.isBusy())return console.warn("[specSim] decode in flight β skipping"),null;t.setBusy(!0);try{const g=[{name:"prose",messages:[{role:"system",content:"You are a helpful assistant."},{role:"user",content:"Write a four-sentence explanation of how photosynthesis works."}]},{name:"code",messages:[{role:"system",content:"You are a helpful coding assistant. Reply with only code, no prose."},{role:"user",content:"Write a TypeScript function that computes the nth Fibonacci number using memoization. Include a short test block that prints fib(10), fib(20), and fib(30)."}]},{name:"summary",messages:[{role:"system",content:"You are a helpful assistant."},{role:"user",content:"List the top five considerations when designing a REST API, with a one-sentence explanation for each."}]}],r=[];for(const e of g){const c=B(o.spec,e.messages,u),s=[];await o.generatePipelined(c,l,d=>{s.push(d)});const a=[...c,...s],m=ye(e.name,a,c.length,i,p);r.push({...m,generatedTokens:s.length}),console.log(`[specSim] ${e.name}: ${s.length} gen tok, hit=${(m.hitRate*100).toFixed(0)}%, Ξ±=${(m.acceptanceRate*100).toFixed(0)}%, accepted/step=${m.meanAcceptedPerStep.toFixed(2)}/${p}, theoretical speedup=${m.theoreticalSpeedup.toFixed(2)}Γ`)}return console.table(r.map(e=>({prompt:e.name,gen:e.generatedTokens,"hit %":+(e.hitRate*100).toFixed(1),"accept Ξ±":+(e.acceptanceRate*100).toFixed(1),"accepted/step":+e.meanAcceptedPerStep.toFixed(2),speedup:+e.theoreticalSpeedup.toFixed(2)}))),r}finally{t.setBusy(!1),t.onIdle()}},window.benchPrefill=async(l=800,i=3)=>{var p;if(t.isBusy())return console.warn("[benchPrefill] decode in flight β skipping"),null;t.setBusy(!0);try{const g="Photosynthesis is the process by which green plants convert light energy into chemical energy, storing it in the bonds of glucose molecules that fuel growth, respiration, and reproduction. ";let r=4,e=[];const c=d=>B(o.spec,[{role:"system",content:"You are a helpful assistant."},{role:"user",content:g.repeat(d)+`
+Summarize the passage above in one sentence.`}],u);for(e=c(r);e.length{});const S=performance.now()-b,k=e.length/(S/1e3),x=d===0?"warmup":`run${d}/${i}`;console.log(`[benchPrefill] ${x}: ${S.toFixed(0)} ms to first token = ${k.toFixed(1)} prefill tok/s`),d>0&&s.push({ttftMs:S,tokPerS:k})}const a=s.map(d=>d.tokPerS).sort((d,b)=>d-b),m=a[Math.floor(a.length/2)];return console.log(`[benchPrefill] median: ${m.toFixed(1)} prefill tok/s (${((p=o.getLastPrefill())==null?void 0:p.chunks)??0} chunks/run)`),{promptTokens:e.length,runs:s,median:m,lastPrefill:o.getLastPrefill()}}finally{t.setBusy(!1),t.onIdle()}},window.benchTurns=async(l=120)=>{if(t.isBusy())return console.warn("[benchTurns] decode in flight β skipping"),null;t.setBusy(!0);try{const i="The city library reopened after a two-year renovation that added a glass atrium, a children's wing, quiet study pods, and a rooftop reading garden overlooking the river. ",p=[`Here is some background material: ${i.repeat(9)}
+What changed in the renovation? Answer in detail.`,`More context: ${i.repeat(9)}
+How do the study pods differ from the reading garden?`,"Now, in one short sentence: what overlooks the river?"];o.resetKVTracking();const g=[{role:"system",content:"You are a helpful, concise assistant."}],r=[];for(let e=0;e{s.length===0&&(m=performance.now()-a),s.push(b)}),g.push({role:"assistant",content:u.decode(s)});const d=o.getLastPrefill();console.log(`[benchTurns] turn ${e+1}: prompt ${c.length} tok, reused ${(d==null?void 0:d.reused)??0}, chunks ${(d==null?void 0:d.chunks)??0}, TTFT ${m.toFixed(0)} ms, generated ${s.length}`),r.push({turn:e+1,promptTokens:c.length,ttftMs:m,...d})}return r}finally{t.setBusy(!1),t.onIdle()}},window.checkReuse=async()=>{var l;if(t.isBusy())return console.warn("[checkReuse] decode in flight β skipping"),null;t.setBusy(!0);try{o.resetKVTracking();const i=[{role:"system",content:"You are a helpful, concise assistant."},{role:"user",content:"Name three primary colors."}],p=B(o.spec,i,u),g=[];await o.generatePipelined(p,48,a=>g.push(a)),i.push({role:"assistant",content:u.decode(g)}),i.push({role:"user",content:"Which of those is the warmest?"});const r=B(o.spec,i,u),e=await o.debugCompareReuse(r),c=(((l=o.getLastPrefill())==null?void 0:l.chunks)??0)>0,s=e.maxAbsDiff===0?" (bit-exact β)":c?" (nonzero: prefix built by CHUNKED prefill vs per-token replay β kernel-variant numerics; rerun with ?chunk=0 for the bit-exact assertion)":" (MISMATCH β reuse rules broken!)";return console.log(`[checkReuse] prompt ${e.promptLen} tok, reused prefix ${e.startPos} β logits max|Ξ| = ${e.maxAbsDiff} mean|Ξ| = ${e.meanAbsDiff}`+s),{...e,chunkedPrefix:c}}finally{t.setBusy(!1),t.onIdle()}},console.log("[bench] harness ready β call `await bench(128, 3)` or `await bench(0, 0, true)` for per-kernel profile"),console.log("[bench] prefill: `await benchPrefill(800, 3)` Β· multi-turn TTFT: `await benchTurns()` Β· reuse assertion: `await checkReuse()`"),console.log("[bench] batched primitive: `await benchBatched(4, 500)` β ffnDown weight-reuse falsifiability test"),console.log("[bench] PLD acceptance sim: `await specSim(160, 3, 3)` β measures spec-decode upside without GPU changes")}const F=t=>document.getElementById(t);function ne(t){const o=F("stats");o&&(o.textContent=t)}function Le(t,o){const u=F("loading-error");if(u){if(u.textContent=t,o){const l=document.createElement("button");l.type="button",l.id="retry-download-btn",l.textContent="Retry download",l.style.cssText="display:block;margin-top:0.6rem;padding:0.45rem 1.1rem;background:var(--accent);color:#000;font-weight:600;font-size:0.8rem;border:none;border-radius:6px;cursor:pointer",l.addEventListener("click",()=>{u.classList.remove("visible"),u.replaceChildren(),o()},{once:!0}),u.appendChild(l)}u.classList.add("visible")}}function Ee(t){const{spec:o}=t,u=t.search??location.search,l=t.poolSlots??0;return pe({spec:o,...l?{expertPool:l}:{},optionalFeatures:["subgroups","timestamp-query","chromium-experimental-subgroup-matrix"],probeSubgroups:!0,onDeviceLost:t.onDeviceLost,buildEngine:({device:i,weights:p,sgSizeOk:g,spec:r})=>{const e=he(u,{hasSubgroupsFeature:i.features.has("subgroups"),sgSizeOk:g}),c=!r.qkNorm&&r.weightFormat!=="mlx-safetensors",s=e.int8KV&&!r.mla;e.int8KV&&!s&&O(`?kv8=1 ignored for ${r.id} β int8 KV does not cover MLA, which caches a latent rather than per-head K/V`);const a=Math.round(r.maxContext*r.kvBytesPerToken/(1024*1024)/(s?2:1));O(`Allocating ${s?"int8 ":""}KV cache β ${r.maxPages} pages Γ ${r.pageSize} = ${r.maxContext} tokens Β· ~${a} MiB`);const m=s?ce(i,r):de(i,r);O("Building decode engineβ¦");const d=new URLSearchParams(u),b=(k,x)=>{const y=d.get(k);if(y===null)return x;const f=Number(y);return Number.isFinite(f)?f:x},S=b("temp",0);return S>0&&O(`Sampling: temperature ${S}, top-p ${b("topp",1)}, min-p ${b("minp",0)}`),ue(i,p,m,{variants:e,fused:c,int8KV:s,spec:r,...l?{expertPool:l}:{},prefixReuse:d.get("reuse")!=="0",chunkedPrefill:d.get("chunk")!=="0",pooledChunkedPrefill:d.get("chunk")==="1",sampling:S>0?{temperature:S,topP:b("topp",1),minP:b("minp",0),seed:b("seed",0)}:void 0})},warmup:async(i,p,g)=>{const r=B(o,[{role:"user",content:"Hi."}],p),e=o.moe?r.slice(0,1):r,c=performance.now();g==null||g(`Warming up pipeline β ${e.length} token(s)${o.moe?" (single-pass: MoE)":""}β¦`),await i.generatePipelined(e,1,()=>{}),g==null||g(`Warmup done in ${((performance.now()-c)/1e3).toFixed(1)}s`)}})}function Ie(t){var J;const{spec:o,tokenizer:u,engine:l}=t,i=we(o),p=t.log??O,g=t.onPhase??(()=>{});fe(i.name,me(o));const r=F("composer"),e=F("inp"),c=F("btn"),s=F("stop-btn"),a=F("new-chat-btn"),m=F("suggest-grid"),d=F("ctx-hint"),b=F("welcome");if(!r||!e||!c||!s){console.error("[zero-tvm] chat UI elements missing");return}const S="You are a helpful, concise assistant. Use Markdown (numbered lists, **bold**, and fenced ```code``` blocks with a language tag) when it clarifies the answer.\n\nYou are running inside zero-tvm (zerotvm.com): an LLM inference engine written by hand in WGSL and TypeScript, running entirely in this browser tab on the user's own GPU through WebGPU β no server, no WebLLM, no TVM. Model weights download once from Hugging Face and cache on this device; prompts, generated tokens and the KV cache never leave the machine. The whole forward pass is 10 hand-written WGSL kernel roles, readable end-to-end. If asked what zero-tvm is, answer from this paragraph.",k=[{role:"system",content:S}];let x=!1,y=!1;const f=`zt-chat-${o.id}`;function w(){try{const n=k.filter(h=>h.role!=="system").slice(-40);localStorage.setItem(f,JSON.stringify(n))}catch{}}function T(){let n=[];try{n=JSON.parse(localStorage.getItem(f)??"[]")}catch{return}!Array.isArray(n)||n.length===0||(n.forEach((h,v)=>{var L;if(h.role==="user"&&typeof h.content=="string")k.push({role:"user",content:h.content}),ee(h.content);else if(h.role==="assistant"&&typeof h.content=="string"){const M={role:"assistant",content:h.content,ids:Array.isArray(h.ids)?h.ids:void 0};k.push(M),te().finish({fullText:h.content,tokens:((L=M.ids)==null?void 0:L.length)??0,tokPerS:0,...v===n.length-1?{onRegenerate:()=>{j()}}:{}})}}),C(B(o,k,u).length))}function $(){!e||!c||(c.disabled=x||e.value.trim().length===0)}function P(n){!e||!c||!s||(x=n,c.hidden=n,s.hidden=!n,s.disabled=!n,e.disabled=n,a&&(a.disabled=n),e.placeholder=n?"Generatingβ¦":`Ask ${i.name} anythingβ¦`)}function C(n){if(d)if(!n)d.textContent="Zero TVM Β· 10 WGSL kernel roles Β· β/β for commands";else if(n>=l.maxContext)d.textContent=`Context full β ${l.maxContext} / ${l.maxContext} tokens Β· start a new chat`;else{const h=Math.round(n/l.maxContext*100);d.textContent=`Context ${n} / ${l.maxContext} tokens (${h}%)`}}function A(){if(x)return;k.length=0,k.push({role:"system",content:S});try{localStorage.removeItem(f)}catch{}const n=F("messages");n&&n.replaceChildren(),b==null||b.classList.remove("hidden"),ne(""),_("Ready","ready"),C(),matchMedia("(pointer: coarse)").matches||e==null||e.focus()}e.disabled=!1,e.placeholder=`Ask ${i.name} anythingβ¦`,a&&(a.disabled=!1),Y(e),$(),C(),ge(),T();const R=!matchMedia("(pointer: coarse)").matches;R&&e.focus(),(J=t.lock)==null||J.onChange(n=>{x||(e.disabled=n,c.disabled=n||e.value.trim().length===0,e.placeholder=n?"Serving a room guestβ¦":`Ask ${i.name} anythingβ¦`)}),e.addEventListener("input",()=>{Y(e),$()}),e.addEventListener("keydown",n=>{n.key==="Enter"&&!n.shiftKey&&(n.preventDefault(),c.disabled||r.requestSubmit())}),r.addEventListener("submit",n=>{n.preventDefault(),c.disabled||se()}),s.addEventListener("click",()=>{y=!0,s.disabled=!0}),a==null||a.addEventListener("click",A),m==null||m.addEventListener("click",n=>{const h=n.target.closest(".suggest");if(!h)return;const v=h.dataset.prompt;v&&(e.value=v,Y(e),$(),r.requestSubmit())});async function D(n,h,v,L){const M=l.maxContext-h.length;p(`Prompt: ${h.length} tokens Β· reply budget ${M} tokens`),C(h.length);const E=v.length>0;E||n.showThinking(),g(E?"generating":"thinking");const K=performance.now();let I=0,V=!0;const H=v.slice();let W=E?u.decode(H):"";try{await l.generatePipelined(h,M,N=>{var X;(X=t.onToken)==null||X.call(t),V&&(E||n.body.replaceChildren(n.cursor),V=!1,g("generating")),I++,H.push(N),W=u.decode(H),n.render(W);const le=(performance.now()-K)/1e3;ne(`${I} tok Β· ${(I/le).toFixed(1)} tok/s`)},()=>y)}catch(N){W+=`
+
+_[Error: ${N}]_`,p(`Error: ${N}`)}const re=(performance.now()-K)/1e3,ae=I/Math.max(re,.001),Z=h.length+I,Q=I>=M,ie=y&&!Q;return n.finish({fullText:W,tokens:H.length,tokPerS:ae,onRegenerate:()=>{j()},notice:Q?{text:`Cut off β the ${l.maxContext}-token context window is full. Start a new chat to keep going.`}:ie?{text:"Stopped β this reply is incomplete.",onContinue:L}:void 0}),g("idle"),C(Z),Z>=l.maxContext&&(_("Context full","error"),p(`Context window full (${l.maxContext} tokens) β start a new chat to continue.`)),{text:W,ids:H}}async function z(){var M,E,K,I;P(!0),y=!1,await((M=t.lock)==null?void 0:M.acquire()),(E=F("messages"))==null||E.querySelectorAll(".truncation-btn").forEach(V=>V.remove());const n=te(),h=B(o,k,u);if(h.length>=l.maxContext){const V=`Context full β the conversation (${h.length} tokens) exceeds the ${l.maxContext}-token window. Start a new chat.`;n.finish({fullText:`_${V}_`,tokens:0,tokPerS:0}),_("Context full","error"),C(h.length),p(V),(K=t.lock)==null||K.release(),P(!1),y=!1,$();return}const v={role:"assistant",content:""};k.push(v);const L=await D(n,h,[],()=>{G(v,n)});v.content=L.text,v.ids=L.ids,w(),(I=t.lock)==null||I.release(),P(!1),y=!1,$(),R&&(e==null||e.focus())}async function G(n,h){var M,E;if(x||!n.ids||k[k.length-1]!==n)return;P(!0),y=!1,await((M=t.lock)==null?void 0:M.acquire());const v=B(o,k.slice(0,-1),u).concat(n.ids),L=await D(h,v,n.ids,()=>{G(n,h)});n.content=L.text,n.ids=L.ids,w(),(E=t.lock)==null||E.release(),P(!1),y=!1,$(),R&&(e==null||e.focus())}const U={new:()=>A(),canvas:()=>{ke()||oe("No runnable code block yet β ask for html, svg, or js.")},...t.commands};function oe(n){var v;const h=document.createElement("div");h.className="sys-note",h.textContent=n,(v=F("messages"))==null||v.appendChild(h)}async function se(){if(x||!e)return;const n=e.value.trim();if(!n)return;const h=n.startsWith("/")?n.slice(1).split(/\s/)[0].toLowerCase():"";if(h&&U[h]){e.value="",Y(e),$(),U[h]();return}e.value="",Y(e),ee(n),k.push({role:"user",content:n}),w(),await z()}async function j(){var h;if(x||k.length===0||k[k.length-1].role!=="assistant")return;k.pop(),w();const n=(h=F("messages"))==null?void 0:h.querySelectorAll(".msg.ai");n&&n.length>0&&n[n.length-1].remove(),await z()}ve({engine:l,tokenizer:u,isBusy:()=>x,setBusy:P,onIdle:$})}export{Ee as b,Le as s,Ie as w};
diff --git a/assets/chat-ui-CQAxV1po6S.js b/assets/chat-ui-CQAxV1po6S.js
new file mode 100644
index 0000000000000000000000000000000000000000..601f5c96e75d8a0ebae41c18326ea3a7b8226482
--- /dev/null
+++ b/assets/chat-ui-CQAxV1po6S.js
@@ -0,0 +1,23 @@
+import{_}from"./preload-helper-ckwbz45pO3.js";const x='',H='',q='';function M(t,s){t.innerHTML=`${x}Copy`,t.addEventListener("click",async()=>{try{await navigator.clipboard.writeText(s()),t.classList.add("copied"),t.innerHTML=`${H}Copied`,setTimeout(()=>{t.classList.remove("copied"),t.innerHTML=`${x}Copy`},1500)}catch{}})}function u(t,s){const n=/(```)|(`[^`]+`)|(\*\*[^*]+\*\*)|(\*[^*\n]+\*)|\[([^\]]+)\]\(([^)\s]+)\)/g;let e=0,c;for(;(c=n.exec(t))!==null;)if(!c[1]){if(c.index>e&&s.appendChild(document.createTextNode(t.slice(e,c.index))),c[2]){const o=document.createElement("code");o.textContent=c[2].slice(1,-1),s.appendChild(o)}else if(c[3]){const o=document.createElement("strong");o.textContent=c[3].slice(2,-2),s.appendChild(o)}else if(c[4]){const o=document.createElement("em");o.textContent=c[4].slice(1,-1),s.appendChild(o)}else if(c[5]&&c[6])if(/^https?:\/\//i.test(c[6])){const o=document.createElement("a");o.href=c[6],o.textContent=c[5],o.target="_blank",o.rel="noopener noreferrer",s.appendChild(o)}else s.appendChild(document.createTextNode(c[0]));e=c.index+c[0].length}eimport("./feats-dlBfBxqpI8.js"),[]).then(a=>a.recordFeat("canvas")).catch(()=>{}),(r=document.getElementById("code-canvas"))==null||r.remove();const n=document.createElement("div");n.id="code-canvas",n.innerHTML=`
+
Anyone with this link chats with the model running in THIS tab,
+ over an encrypted peer-to-peer data channel. Keep the tab open and the
+ machine awake; close the tab to end the room.
+ Runs the full Zero-TVM forward pass on a battery of diverse prompts and
+ shows the top-10 next tokens + a short greedy continuation for each.
+ Pair this with the kernel suites in the repo (npm run test:kernels)
+ to check that the live chat path generalizes beyond a single
+ memorized prompt.
+
+
+
~2 GB Phi-3 weights download on first run, cached after
+
+
+
Initializing...
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/weights-cache-sw.js b/weights-cache-sw.js
new file mode 100644
index 0000000000000000000000000000000000000000..b4ead6e97613d90104b9f761452858112e89f3ae
--- /dev/null
+++ b/weights-cache-sw.js
@@ -0,0 +1,104 @@
+/* eslint-env serviceworker */
+/**
+ * weights-cache-sw.js β Phi-3 weight cache, shared by any page that loads those weights.
+ *
+ * Both zero-tvm's hand-written loader and WebLLM fetch the same files from the
+ * HF mirror at
+ * `huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC/resolve/main/*`.
+ * Without this SW each cache their own copy, so visiting both pages downloads
+ * 1.8 GB twice. This SW intercepts those URLs and serves from a shared OPFS
+ * directory (`zero-tvm-weights/`, the same dir zero-tvm's own loader uses for
+ * its fast path) β populating it on first download from either side.
+ *
+ * Result: whichever page the visitor opens first pays the network cost once.
+ * Subsequent visits to either page hit OPFS instantly.
+ */
+
+// KEEP IN SYNC with WEIGHTS_OPFS_DIR in src/zero-tvm/weight-loader.ts β this
+// SW is plain JS and cannot import the exported constant, so the two
+// definitions are paired by hand.
+const SHARED_DIR = "zero-tvm-weights";
+const HF_PHI3_HOST = "huggingface.co";
+const HF_PHI3_PATH_RE = /^\/mlc-ai\/Phi-3-mini-4k-instruct-q4f16_1-MLC\/resolve\/[^/]+\//;
+
+self.addEventListener("install", () => self.skipWaiting());
+self.addEventListener("activate", (e) => e.waitUntil(self.clients.claim()));
+
+self.addEventListener("fetch", (event) => {
+ const req = event.request;
+ if (req.method !== "GET") return;
+ let url;
+ try { url = new URL(req.url); } catch { return; }
+ if (url.host !== HF_PHI3_HOST || !HF_PHI3_PATH_RE.test(url.pathname)) return;
+
+ event.respondWith(handle(req, url));
+});
+
+async function handle(req, url) {
+ // Flatten the path-suffix into an OPFS-safe filename, matching the
+ // `opfsKey()` convention in zero-tvm's weight-loader.ts byte-for-byte
+ // (so files written by the SW are reused by zero-tvm's direct loader,
+ // and vice versa).
+ const dataPath = url.pathname.replace(HF_PHI3_PATH_RE, "");
+ const key = dataPath.replace(/[^A-Za-z0-9._-]/g, "_");
+
+ // 1. Try OPFS hit first.
+ const cached = await opfsRead(key).catch(() => null);
+ if (cached) {
+ return new Response(cached, {
+ status: 200,
+ headers: {
+ "content-type": guessType(key),
+ "content-length": String(cached.byteLength),
+ "x-zero-tvm-cache": "opfs-hit",
+ },
+ });
+ }
+
+ // 2. Network fall-through. Stream into OPFS so subsequent visits hit cache.
+ const netResp = await fetch(req);
+ // Only cache complete 200 bodies: a 206 (the weight-loader resumes dead
+ // transfers with Range requests) buffered under the full key would poison
+ // the shared cache with a truncated shard. Pass everything else through.
+ if (netResp.status !== 200 || !netResp.body) return netResp;
+
+ const buf = await netResp.clone().arrayBuffer();
+ // Best-effort write β failures (quota, no OPFS) are fine; just no caching.
+ opfsWrite(key, buf).catch(() => {});
+
+ return new Response(buf, {
+ status: netResp.status,
+ headers: {
+ "content-type": netResp.headers.get("content-type") || guessType(key),
+ "content-length": String(buf.byteLength),
+ "x-zero-tvm-cache": "network",
+ },
+ });
+}
+
+async function getDir() {
+ if (!self.navigator?.storage?.getDirectory) throw new Error("no OPFS in SW");
+ const root = await self.navigator.storage.getDirectory();
+ return root.getDirectoryHandle(SHARED_DIR, { create: true });
+}
+
+async function opfsRead(key) {
+ const dir = await getDir();
+ const fh = await dir.getFileHandle(key);
+ const file = await fh.getFile();
+ return await file.arrayBuffer();
+}
+
+async function opfsWrite(key, buf) {
+ const dir = await getDir();
+ const fh = await dir.getFileHandle(key, { create: true });
+ const writer = await fh.createWritable();
+ await writer.write(buf);
+ await writer.close();
+}
+
+function guessType(key) {
+ if (key.endsWith(".json")) return "application/json";
+ if (key.endsWith(".wasm")) return "application/wasm";
+ return "application/octet-stream";
+}
diff --git a/zero-tvm.html b/zero-tvm.html
new file mode 100644
index 0000000000000000000000000000000000000000..efe0ebb14c1c1d84b7544f0f9dd89cb38ef35692
--- /dev/null
+++ b/zero-tvm.html
@@ -0,0 +1,737 @@
+
+
+
+
+
+ Zero-TVM Chat β LLMs up to a 35B MoE on hand-written WebGPU kernels
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+