README.md

---
title: Bonsai Chat · WebGPU
emoji: 🌱
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
suggested_hardware: cpu-basic
fullWidth: true
short_description: Private 1-bit Bonsai chat, fully local in your browser
models:
- WaveCut/Bonsai-web-GGUF
---

# Bonsai WebGPU Chat

Private-by-construction, on-device chat with Prism ML's end-to-end 1-bit
Bonsai models. Model weights are downloaded directly into browser storage;
prompt processing and generation run inside this tab through wllama and
llama.cpp WebAssembly/WebGPU.

## Runtime tiers

- **Bonsai 1.7B** — fastest download and broadest CPU fallback.
- **Bonsai 4B** — balanced local chat.
- **Bonsai 8B** — primary agentic/tool model.
- **Bonsai 27B** — WebGPU-only hybrid model with strict adapter and full-graph
  checks; the UI refuses silent CPU fallback.

The chat UI allocates a 4,096-token context and requests up to 4,096 completion
tokens by default. Prompt text, system instructions, tool schemas, prior turns,
and generated tokens share that same context window, so 4,096 is a completion
ceiling rather than a guarantee after a non-empty prompt.

The Space manifest pins
[`WaveCut/Bonsai-web-GGUF`](https://huggingface.co/WaveCut/Bonsai-web-GGUF)
to release commit `d85382fa09fe868c0242d81488dfc2edd8d3729b` (tag
`v1.1.3`); its published release manifest separately pins the immutable
weight commit `112ea7a1a6229bde132b176b9a72477a7ecfde64`. Every shard has a recorded
byte size and SHA-256.

The repository includes the verified `dist/` production bundle. The CPU Basic
Docker Space serves that bundle directly with Nginx and does not run npm or
Vite during its container build. The checked-in bundle is produced locally by
`npm ci && npm run build`; source, tests, lockfile, and exact runtime assets
remain in the same Space repository.

## Chat workspace

Chat sessions and their system instructions are saved locally in IndexedDB.
The chat list can reopen, continue, or delete a session without touching the
model cache. New chats begin with the bundled private-assistant instructions;
an empty chat exposes them directly, while an active chat keeps them in Chat
settings. Appearance can follow the operating system or be pinned to the light
or dark theme. Both sidebars can collapse, and the transcript can switch
between centered and full-width layouts.

Every user and assistant message exposes copy and branch-aware regenerate
controls. Regenerating a user turn discards the later local branch and answers
that turn again; regenerating an assistant turn replaces that response and its
later branch. Fenced code is highlighted incrementally with the pinned Shiki
streaming tokenizer, including incomplete fences while generation is active.

## Browser requirements

Chrome or Edge with WebGPU is recommended. Safari uses the stable compatibility
runtime but is substantially slower, so the UI calls this limitation out before
the model is loaded. CPU-WASM fallback requires a
cross-origin-isolated browser with `SharedArrayBuffer`; the Nginx configuration
enables it. Model storage uses the browser cache/OPFS path exposed by wllama and
requests persistent storage before the first model byte is downloaded. A browser
denial is non-fatal, and the storage panel lets the operator request it again.

The Chrome/Edge JSPI release uses the validated raw-layout vec4 Q1_0 WG64
matvec kernel. Safari keeps the scalar Q1_0 compat/Asyncify runtime; the two
paths have independent artifact hashes and the browser selects between them
automatically.

Every fresh or cached shard is accepted only after an incremental SHA-256 over
its actual bytes. Hashing reads `Blob.stream()` and does not buffer a whole
shard in JavaScript memory. A failed shard is removed and a retry restarts that
shard from byte zero; the UI does not claim resumable Range downloads.

27B is intentionally gated. It is unavailable when the adapter's
`maxStorageBufferBindingSize`, `shader-f16`, browser quota, or all-WebGPU graph
requirement does not pass. Chrome on Apple Silicon is the validated release
path. Safari 26.5 on Apple Silicon has also passed a real foreground cold 27B
load through compat/Asyncify plus explicit OPFS eviction. The normal compat
path caps prefill at `n_batch=32` and `n_ubatch=16` so Safari submits shorter
Metal command buffers; explicit benchmark tuning can still override the cap.
On Safari 26.5.2, the final 64-token gate completed at 1.2 prefill tok/s and
3.5 decode tok/s with one graph, zero CPU ops, and 65/65 GPU layers. Chrome's
JSPI path does not use this cap. Firefox gating was
validated on macOS: it selected compat, offered CPU-WASM for dense tiers, and
refused 27B without WebGPU; Firefox model generation was not exercised.
Windows Chrome/Edge on a discrete GPU and a physical low-limit iGPU laptop
remain explicit release-matrix gaps. Dense CPU fallback is offered only when
the browser exposes the required cross-origin-isolated WASM features.

## Browser benchmark

[`?view=bench`](https://wavecut-bonsai-chat-webgpu.hf.space/?view=bench)
is a separate, shareable benchmark surface rendered by the same browser app.
The query deep-link keeps the benchmark executable under the same COOP/COEP
headers. It loads nothing until **Run benchmark** is pressed, then records first-load and
verified-cache reload wall time, TTFT,
prefill/decode throughput, graph placement, device limits, engine revisions,
the exact runtime artifact hash, and a fixed temperature-0 prompt. Schema v3
exports the exact messages, sampling controls, raw generated text, pinned model
source/shard hashes, and an opt-in sampled-token trace. Normal chat does not
request logprobs and returns no token-id trace.

The default `field-core-v1` workload remains short. The separate
`state-drift-1k-v1` evidence workload locks greedy sampling and refuses to
export unless all 1,024 sampled token IDs and the 64/128/256/512/768/1,024
uint32-LE prefix hashes are present. Its diagnostic-only 27B WebGPU path also
validates the pinned native CPU prompt/reference fixture, then teacher-forces
that fixed 1,024-token sequence through raw token-ID prefixes. Schema v3 keeps
the forced reference logprob, natural top five, mean NLL, and perplexity
separate from the natural browser generation. This scoring method is not
available to normal chat.

On the pinned Apple `metal-3` validation path, the current JSPI 1,024-token
27B run reached 12.1 tok/s with one graph, zero CPU ops, and 65/65 GPU layers.
The exact-current scalar control reached 10.25 tok/s. Both arms returned the
same 1,024 token IDs, the same top-five candidate IDs at every position, and
the same visible text; the promoted vec4 arm improved decode by 18.05%.
Exact CPU/browser token parity remains `NO-GO` because the first natural
mismatch is a reciprocal 13/198 top-one/top-two flip at position 30. The
separate numerical gate passes: browser and native CPU mean NLL are
0.0456365662 and 0.0455182718 (absolute delta 0.0001182944 against a 0.01
limit), with near-tie margins 0.006769 and 0.003605. This supports a bounded
GPU floating-point tolerance; it is not a bit-exact parity claim. That run
uses the exact JSPI and patch hashes published below; the release does not
claim bit-exact browser/native parity.

`n_batch`, `n_ubatch`, Flash Attention, paired K/V cache types, and the pinned
JSPI/compat WASM flavor are available there as explicitly experimental
controls. The browser report separates requested and native-observed values,
records the actually loaded artifact hashes, and fails the run when they
disagree. A pinned local 1.7B WebGPU sweep proved Flash off/auto with F16, Q8_0,
and Q4_0 KV: every row retained one graph, 29/29 GPU layers, zero CPU ops, and
the same short deterministic output hash. Observed KV allocation fell from
112 MiB for F16 to 59.5 MiB for Q8_0 and 31.5 MiB for Q4_0.

The separate final-artifact 27B >8K smoke used an 8,448-token context with
8,314 prompt tokens, Flash auto, and Q4_0/Q4_0 KV. It measured 7.55 prefill
tok/s and 6.42 decode tok/s, retained one graph with zero CPU ops and 65/65
GPU layers, and exported all 8 requested top-five token records. This is one
local Apple `metal-3` result; it does not establish a cross-device default.

A same-Chrome A/B with one warmup and three alternating measured runs per arm
also proved explicit JSPI/compat switching. Median warm load was 2,257.7 ms for
JSPI and 2,799.1 ms for compat; all eight outputs and graph tripwires matched.
These are experimental single-device results, not release recommendations;
normal chat keeps Flash off, F16/F16, and automatic WASM selection.

The historical public v1.1.2 prewarmed-cache baseline below is one Apple
`metal-3` / headless Chrome sample, not a cross-device study and not a
first-download claim:

| Tier | Backend | Load | TTFT | Decode |
| --- | --- | ---: | ---: | ---: |
| 1.7B | WebGPU | 3,314 ms | 300 ms | 71.9 tok/s |
| 1.7B | CPU-WASM | 3,844 ms | 6,092 ms | 10.6 tok/s |
| 8B | WebGPU | 9,870 ms | 998 ms | 33.8 tok/s |
| 27B | WebGPU | 29,488 ms | 3,348 ms | 11.9 tok/s |

## Local tools and safety boundary

Tools are off by default. Memory writes/deletes require operator confirmation.
`js_eval` runs in a disposable opaque-origin Worker with network and browser
storage blocked, a 5-second timeout, and bounded output. HTML artifacts are
static HTML/CSS previews: their iframe has no sandbox permissions, scripts do
not run, and network requests are blocked. Prompt, token, tool, and memory data
stay in the browser; only model/static asset downloads contact Hugging Face.

## Provenance

- llama.cpp baseline: `00fa7cb284cbf133fc426733bd64238a3588a33e`
- Dawn native WebGPU validation: `18eb229ef5f707c1464cc581252e7603c73a3ef0`
- custom wllama source: `912c18b75d4358c1405a64646b8dbe43a205943b`
- custom nested llama.cpp: `00fa7cb284cbf133fc426733bd64238a3588a33e`
- custom patch set: `0a1d7ac5b0edb62cdb8ff5e5e0005c07b5a2fb7a91059f678c49d973fa456949`
- custom ESM: `9a3d15ca29924927b6e95971c5b355f54b581f74089bd74975a063416f7fc65d`
- JSPI WASM: `93ebb472969c4ccfe9d406e2ebd043da288a6d513a180a57827211278f4c82ed`
- compat WASM: `91f1566e3dfa4cd591d902c03bfc2c92c9ceef8d5ce93cecae197ee1f4f5dd5d`
- compat worker: `0b667db536815fb9e5dc9ffbfbd6ba29affdda38a4ba892d5372ddb41ce9a8c8`
- JSPI Q1_0 profile: raw-layout vec4 WG64; Safari compat Q1_0 profile: scalar

The vendored runtime exposes the validated token-embedding WebGPU placement
needed for single-graph Bonsai inference and fails closed when placement or the
backend tripwire disagrees. Its exact JS/WASM hashes are checked during
`postinstall`; source/patch provenance is in
`vendor/wllama-bonsai/SOURCE.json`. Third-party notices are in
`THIRD_PARTY_NOTICES.md` and `public/licenses/`.

The pinned WebGPU source implements Flash Attention with Q4_0/Q8_0 K/V. The
benchmark lane has short 1.7B deterministic-output and memory evidence for
both quantized types, a 27B long-state result only for Flash-off F16, and a
27B >8K smoke only for Flash-auto Q4_0. No 27B Q8_0 long-state result is
claimed. Repeated and cross-device gates remain open, so Flash Attention
remains off and F16/F16 remains the release default.

## Licenses and attribution

The Bonsai GGUF tiers are Apache-2.0 model artifacts; their upstream notices
are published in `WaveCut/Bonsai-web-GGUF@v1.1.3/notices/`. The shipped
wllama/llama.cpp engines, React runtime, `@noble/hashes`, Shiki, and
`@shikijs/stream` are MIT-licensed. Exact runtime license texts are included
in `public/licenses/`; the consolidated inventory is
`THIRD_PARTY_NOTICES.md`.