--- title: Zero-TVM emoji: 🧠 colorFrom: indigo colorTo: purple sdk: static pinned: true license: mit thumbnail: https://huggingface.co/spaces/abgunaydin/zero-tvm/resolve/main/og.png short_description: Phi-3 to a 35B MoE in the browser, on hand-written WGSL tags: - webgpu - llm - inference - on-device - wgsl models: - microsoft/Phi-3-mini-4k-instruct - Qwen/Qwen3-4B - Qwen/Qwen3.5-4B - meta-llama/Llama-3.2-1B-Instruct - mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC - mlc-ai/Qwen3-4B-q4f16_1-MLC - mlc-ai/Qwen3.5-4B-q4f16_1-MLC - mlx-community/Qwen3-4B-4bit - mlx-community/Qwen3-30B-A3B-4bit - mlx-community/Llama-3.2-1B-Instruct-4bit - lmstudio-community/Qwen3.6-35B-A3B-MLX-4bit - abgunaydin/Qwen3.6-35B-A3B-MLX-q3exp --- # Zero-TVM Models from a 3.8B dense to a 35B sparse MoE, running in the browser on hand-written WGSL. No TVM, no ONNX, no WASM runtime. The standard way to run a modern LLM in a browser is [WebLLM / MLC-LLM](https://webllm.mlc.ai/), which ships an Apache-TVM compiler pipeline that autotunes its WGSL β€” 85 shaders captured across a session, around 11 of them on the decode path. This Space replaces that entire stack with **10 kernel roles** of hand-written WGSL β€” and runs models WebLLM ships (same weights, measured faster) as well as one it does not: Qwen3.6-35B-A3B, a 256-expert sparse MoE. The whole forward pass β€” 32 transformer layers, paged KV cache, int4-dequant matmul, RoPE, fused FFN, RMSNorm, paged attention, argmax sampling β€” is readable end-to-end in a single sitting. That is the point. ## Headline result Measured on an Apple M2 Max (Chrome 150, WebLLM v0.2.80, identical Phi-3-mini-q4f16_1 weights, same-session head-to-head, 2026-07-30). Two numbers, always both: **total** is wall-clock throughput with prefill included (both engines doing identical work), **decode** excludes prefill. | | WebLLM (TVM) | Zero-TVM | |---|---|---| | WGSL kernels (decode path) | **~11 TVM-generated** (85 captured across a session) | **10 hand-written roles** | | Dispatches per decode step | **342** | **260** default (228 with `?splitk=0`) | | Total throughput (prefill + decode) | **59.95 tok/s** | **69.55 tok/s** β€” **+16.0%** | | Decode only | **63.23 tok/s** (self-reported) | **83.10 tok/s** β€” **+31.4%** | | Time to first token (~35-tok prompt) | **~150 ms** (implied) | **291 ms** β€” *WebLLM wins this one* | | JS bundle (excl. weights) | **~6.0 MB** / ~2.2 MB gz | **~460 kB** / ~126 kB gz | **The hand-written kernels don't just keep up with the autotuning compiler β€” on this hardware they beat it on throughput.** For decoder-only LLMs of this shape, most of the compiler's complexity budget isn't buying much β€” the expensive parts are matmul, attention, and int4 dequant. Everything else is plumbing, and ten hand-written roles cover it. **The gap grows with how recent the architecture is** β€” +16% / +31% (total / decode) on 2024's Phi-3, +32% / +58% on 2025's Qwen3-4B, +100% / +114% on 2026's Qwen3.5-4B. Consistent with compiler stacks having had less time to tune newer architectures; an observation across three points on one machine, not a proven law. **Honest negative, published not buried:** WebLLM reaches the first token faster than we do on short prompts (~150 ms implied vs our 291 ms on Phi-3; 453 ms on Qwen3-4B). We win sustained throughput and lose the first-token sprint on short inputs β€” our chunked prefill is strong on long ones (202 tok/s at 816 tokens) and prefix reuse makes follow-up turns free. Exact medians, full methodology, and the optimization experiments that were measured and *dropped* live in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md). (An earlier measurement on an M2 Pro with an older, buggier engine read 22% *behind* β€” that history is preserved there too, not retconned. So are two 2026-07-29 Qwen pairs that a bench-harness defect made non-comparable; they are marked withdrawn with the reason, not deleted.) ## Run it The Space opens on a character-select entrance: pick a model and it loads and chats in place. [`zero-tvm.html`](./zero-tvm.html) is the same chat as a direct link, if you want to deep-link a specific model. Picking a character starts the download immediately β€” the sheet shows the weight size before you click, and there is no second confirmation. It streams ~2 GB of Phi-3-mini-q4f16_1 weights from the Hugging Face mirror at [`mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC) into OPFS (Origin Private File System). Subsequent loads are instant β€” the weights stay in OPFS, so returning to a model you have already downloaded starts it straight away. **Requirements**: Chrome / Edge with WebGPU enabled and the `shader-f16` feature available (default on macOS Apple-Silicon, enabled on most modern Windows / Linux GPUs). ## Models All three pairs re-measured 2026-07-30 under the corrected protocol β€” same session, identical local weight bytes, both engines paying a full prefill on every run, total and decode reported side by side. - **Phi-3-mini-4k-instruct (3.8B, q4f16_1)** β€” the default; every headline number above is Phi-3, no URL flag needed. Zero-TVM **69.55 tok/s total** (TTFT 291 ms, decode 83.10) vs WebLLM 0.2.80's **59.95 tok/s total** (self-reported decode 63.23) β€” **+16.0% total, +31.4% decode**. - **Qwen3-4B (q4f16_1)** β€” append `?model=qwen3` to `zero-tvm.html` or `validate.html`; weights stream from [`mlc-ai/Qwen3-4B-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Qwen3-4B-q4f16_1-MLC) (~2.3 GB). A port on the spec-parameterized engine: GQA 32/8, per-head QK-norm, byte-level BPE, tied lm_head, with the tuning round's fused post-matmul chain (`qk_norm_rope_append`, 8 dispatches/layer) and K%512 wide loads. Zero-TVM **59.85 tok/s total** (TTFT 453 ms, decode 75.49) vs WebLLM 0.2.84's prebuilt Qwen3-4B at **45.46 tok/s total** (self-reported decode 47.77) β€” **+31.7% total, +58.0% decode**. This is the model where WebLLM most clearly beats us on first-token latency (~150 ms implied vs our 453 ms). The previously published "75.7 vs 43.8, +73%" pair is **withdrawn** β€” a bench-harness defect had stopped our half paying prefill; and the 2026-07-28 pair (25.43 vs 14.15) was a degraded session that did not reproduce. Both are kept as dated history with the reason in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md). - **Qwen3.5-4B (q4f16_1)** β€” append `?model=qwen35` to `zero-tvm.html` or `validate.html`; weights stream from [`mlc-ai/Qwen3.5-4B-q4f16_1-MLC`](https://huggingface.co/mlc-ai/Qwen3.5-4B-q4f16_1-MLC) (~2.6 GB). The first *hybrid* on the engine: 24 gated-DeltaNet (linear-attention) layers + 8 gated-attention layers (GQA 16/4, head_dim 256, partial RoPE, sigmoid attention gate) β€” to our knowledge the first hand-written-kernel int4 gated-DeltaNet hybrid in a browser. Zero-TVM **65.28 tok/s total** (TTFT 171 ms, decode 73.30) vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at **32.56 tok/s total** (self-reported decode 34.32) β€” **+100.5% total, +113.6% decode**, and the one model where first-token latency is a wash rather than a loss. The GDN decode kernels are still scalar (non-subgroup), so the decode number is a floor; prefill runs in chunks of ≀64 (202 tok/s on an 816-token prompt) and every model reuses its cross-turn prefix (turn-3 first token 14.3 s β†’ 0.19 s, logits verified bit-identical to a fresh prefill). The previously published "65.7 vs 34.0, +93%" cross-check is **withdrawn** (same harness defect); the earlier 53.07/32.36 (+64%) and 47.99/31.99 (+50%) pairs were like-for-like but are superseded. One machine, one pair each; caveats in [BENCH.md](https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md). ## Pages in this Space - [`index.html`](./index.html) β€” the entrance: pick a model, chat in place (start here) - [`zero-tvm.html`](./zero-tvm.html) β€” the chat surface as a direct link - [`validate.html`](./validate.html) β€” multi-prompt smoke test - [`docs.html`](./docs.html) β€” annotated reference, including the kernel walkthrough Six pages were removed on 2026-08-14 β€” `architecture`, `demo`, `compiler-chat`, `dump`, `shaders`, `webllm-bench`. Three of them started multi-gigabyte downloads on page load. `docs.html` carries what the first two explained. ## URL flags `zero-tvm.html` accepts query flags for A/B-ing shader variants without rebuilding: - `?sg=0` β€” disable all subgroup shaders (argmax / attention / QKV matmul) - `?sgqkv=0` / `?sgattn=0` / `?sgargmax=0` β€” disable one at a time - `?qkvtile=1` / `?qkvtile2=1` β€” opt into tiled QKV variants - `?ffnsg=1` β€” opt into the tiled-subgroup fused FFN - `?kv8=1` β€” opt into the int8 KV cache path ## How it relates to a published paper Zero-TVM is the LLM-decoding application of a broader thesis: **per-dispatch overhead is the dominant tax on browser GPU compute, and most workloads collapse from "needs a server" to "runs in a browser tab" once you fuse the loop.** Two preprints document the underlying technique: - [Single-Kernel Fusion for Sequential Fitness Evaluation via WebGPU Compute Shaders](https://doi.org/10.5281/zenodo.19342888) (the foundational result) - [Single-Kernel Fusion for Autoregressive Transformer Decoding via WebGPU Compute Shaders](https://doi.org/10.5281/zenodo.19344277) (the LLM application β€” the work this Space embodies) ## Source / canonical site - **GitHub**: [github.com/abgnydn/zero-tvm](https://github.com/abgnydn/zero-tvm) - **Canonical site**: [zerotvm.com](https://zerotvm.com) - **Research umbrella**: [kernelfusion.dev](https://kernelfusion.dev) - **Companion benchmarks**: [gpubench.dev](https://gpubench.dev) ## License MIT. Reference comparison code (WebLLM compiler-chat path) belongs to its respective authors and is governed by their licenses. ## Author Ahmet Barış GΓΌnaydΔ±n Β· [barisgunaydin.com](https://barisgunaydin.com) Β· senior full-stack consultant by day, independent researcher by night.