README.md

metadata

title: Bonsai Chat · WebGPU
emoji: 🌱
colorFrom: green
colorTo: yellow
sdk: docker
app_port: 7860
suggested_hardware: cpu-basic
fullWidth: true
short_description: Private 1-bit Bonsai chat, fully local in your browser
models:
  - WaveCut/Bonsai-web-GGUF

Bonsai WebGPU Chat

Private-by-construction, on-device chat with Prism ML's end-to-end 1-bit Bonsai models. Model weights are downloaded directly into browser storage; prompt processing and generation run inside this tab through wllama and llama.cpp WebAssembly/WebGPU.

Runtime tiers

• Bonsai 1.7B — fastest download and broadest CPU fallback.
• Bonsai 4B — balanced local chat.
• Bonsai 8B — primary agentic/tool model.
• Bonsai 27B — WebGPU-only hybrid model with strict adapter and full-graph checks; the UI refuses silent CPU fallback.

The chat UI allocates a 4,096-token context and requests up to 4,096 completion tokens by default. Prompt text, system instructions, tool schemas, prior turns, and generated tokens share that same context window, so 4,096 is a completion ceiling rather than a guarantee after a non-empty prompt.

The Space manifest pins WaveCut/Bonsai-web-GGUF to release commit d85382fa09fe868c0242d81488dfc2edd8d3729b (tag v1.1.3); its published release manifest separately pins the immutable weight commit 112ea7a1a6229bde132b176b9a72477a7ecfde64. Every shard has a recorded byte size and SHA-256.

The repository includes the verified dist/ production bundle. The CPU Basic Docker Space serves that bundle directly with Nginx and does not run npm or Vite during its container build. The checked-in bundle is produced locally by npm ci && npm run build; source, tests, lockfile, and exact runtime assets remain in the same Space repository.

Chat workspace

Chat sessions and their system instructions are saved locally in IndexedDB. The chat list can reopen, continue, or delete a session without touching the model cache. New chats begin with the bundled private-assistant instructions; an empty chat exposes them directly, while an active chat keeps them in Chat settings. Appearance can follow the operating system or be pinned to the light or dark theme. Both sidebars can collapse, and the transcript can switch between centered and full-width layouts.

Every user and assistant message exposes copy and branch-aware regenerate controls. Regenerating a user turn discards the later local branch and answers that turn again; regenerating an assistant turn replaces that response and its later branch. Fenced code is highlighted incrementally with the pinned Shiki streaming tokenizer, including incomplete fences while generation is active.

Browser requirements

Chrome or Edge with WebGPU is recommended. Safari uses the stable compatibility runtime but is substantially slower, so the UI calls this limitation out before the model is loaded. CPU-WASM fallback requires a cross-origin-isolated browser with SharedArrayBuffer; the Nginx configuration enables it. Model storage uses the browser cache/OPFS path exposed by wllama and requests persistent storage before the first model byte is downloaded. A browser denial is non-fatal, and the storage panel lets the operator request it again.

The Chrome/Edge JSPI release uses the validated raw-layout vec4 Q1_0 WG64 matvec kernel. Safari keeps the scalar Q1_0 compat/Asyncify runtime; the two paths have independent artifact hashes and the browser selects between them automatically.

Every fresh or cached shard is accepted only after an incremental SHA-256 over its actual bytes. Hashing reads Blob.stream() and does not buffer a whole shard in JavaScript memory. A failed shard is removed and a retry restarts that shard from byte zero; the UI does not claim resumable Range downloads.

27B is intentionally gated. It is unavailable when the adapter's maxStorageBufferBindingSize, shader-f16, browser quota, or all-WebGPU graph requirement does not pass. Chrome on Apple Silicon is the validated release path. Safari 26.5 on Apple Silicon has also passed a real foreground cold 27B load through compat/Asyncify plus explicit OPFS eviction. The normal compat path caps prefill at n_batch=32 and n_ubatch=16 so Safari submits shorter Metal command buffers; explicit benchmark tuning can still override the cap. On Safari 26.5.2, the final 64-token gate completed at 1.2 prefill tok/s and 3.5 decode tok/s with one graph, zero CPU ops, and 65/65 GPU layers. Chrome's JSPI path does not use this cap. Firefox gating was validated on macOS: it selected compat, offered CPU-WASM for dense tiers, and refused 27B without WebGPU; Firefox model generation was not exercised. Windows Chrome/Edge on a discrete GPU and a physical low-limit iGPU laptop remain explicit release-matrix gaps. Dense CPU fallback is offered only when the browser exposes the required cross-origin-isolated WASM features.

Browser benchmark

?view=bench is a separate, shareable benchmark surface rendered by the same browser app. The query deep-link keeps the benchmark executable under the same COOP/COEP headers. It loads nothing until Run benchmark is pressed, then records first-load and verified-cache reload wall time, TTFT, prefill/decode throughput, graph placement, device limits, engine revisions, the exact runtime artifact hash, and a fixed temperature-0 prompt. Schema v3 exports the exact messages, sampling controls, raw generated text, pinned model source/shard hashes, and an opt-in sampled-token trace. Normal chat does not request logprobs and returns no token-id trace.

The default field-core-v1 workload remains short. The separate state-drift-1k-v1 evidence workload locks greedy sampling and refuses to export unless all 1,024 sampled token IDs and the 64/128/256/512/768/1,024 uint32-LE prefix hashes are present. Its diagnostic-only 27B WebGPU path also validates the pinned native CPU prompt/reference fixture, then teacher-forces that fixed 1,024-token sequence through raw token-ID prefixes. Schema v3 keeps the forced reference logprob, natural top five, mean NLL, and perplexity separate from the natural browser generation. This scoring method is not available to normal chat.

On the pinned Apple metal-3 validation path, the current JSPI 1,024-token 27B run reached 12.1 tok/s with one graph, zero CPU ops, and 65/65 GPU layers. The exact-current scalar control reached 10.25 tok/s. Both arms returned the same 1,024 token IDs, the same top-five candidate IDs at every position, and the same visible text; the promoted vec4 arm improved decode by 18.05%. Exact CPU/browser token parity remains NO-GO because the first natural mismatch is a reciprocal 13/198 top-one/top-two flip at position 30. The separate numerical gate passes: browser and native CPU mean NLL are 0.0456365662 and 0.0455182718 (absolute delta 0.0001182944 against a 0.01 limit), with near-tie margins 0.006769 and 0.003605. This supports a bounded GPU floating-point tolerance; it is not a bit-exact parity claim. That run uses the exact JSPI and patch hashes published below; the release does not claim bit-exact browser/native parity.

n_batch, n_ubatch, Flash Attention, paired K/V cache types, and the pinned JSPI/compat WASM flavor are available there as explicitly experimental controls. The browser report separates requested and native-observed values, records the actually loaded artifact hashes, and fails the run when they disagree. A pinned local 1.7B WebGPU sweep proved Flash off/auto with F16, Q8_0, and Q4_0 KV: every row retained one graph, 29/29 GPU layers, zero CPU ops, and the same short deterministic output hash. Observed KV allocation fell from 112 MiB for F16 to 59.5 MiB for Q8_0 and 31.5 MiB for Q4_0.

The separate final-artifact 27B >8K smoke used an 8,448-token context with 8,314 prompt tokens, Flash auto, and Q4_0/Q4_0 KV. It measured 7.55 prefill tok/s and 6.42 decode tok/s, retained one graph with zero CPU ops and 65/65 GPU layers, and exported all 8 requested top-five token records. This is one local Apple metal-3 result; it does not establish a cross-device default.

A same-Chrome A/B with one warmup and three alternating measured runs per arm also proved explicit JSPI/compat switching. Median warm load was 2,257.7 ms for JSPI and 2,799.1 ms for compat; all eight outputs and graph tripwires matched. These are experimental single-device results, not release recommendations; normal chat keeps Flash off, F16/F16, and automatic WASM selection.

The historical public v1.1.2 prewarmed-cache baseline below is one Apple metal-3 / headless Chrome sample, not a cross-device study and not a first-download claim:

Tier	Backend	Load	TTFT	Decode
1.7B	WebGPU	3,314 ms	300 ms	71.9 tok/s
1.7B	CPU-WASM	3,844 ms	6,092 ms	10.6 tok/s
8B	WebGPU	9,870 ms	998 ms	33.8 tok/s
27B	WebGPU	29,488 ms	3,348 ms	11.9 tok/s

Local tools and safety boundary

Tools are off by default. Memory writes/deletes require operator confirmation. js_eval runs in a disposable opaque-origin Worker with network and browser storage blocked, a 5-second timeout, and bounded output. HTML artifacts are static HTML/CSS previews: their iframe has no sandbox permissions, scripts do not run, and network requests are blocked. Prompt, token, tool, and memory data stay in the browser; only model/static asset downloads contact Hugging Face.

Provenance

• llama.cpp baseline: 00fa7cb284cbf133fc426733bd64238a3588a33e
• Dawn native WebGPU validation: 18eb229ef5f707c1464cc581252e7603c73a3ef0
• custom wllama source: 912c18b75d4358c1405a64646b8dbe43a205943b
• custom nested llama.cpp: 00fa7cb284cbf133fc426733bd64238a3588a33e
• custom patch set: 0a1d7ac5b0edb62cdb8ff5e5e0005c07b5a2fb7a91059f678c49d973fa456949
• custom ESM: 9a3d15ca29924927b6e95971c5b355f54b581f74089bd74975a063416f7fc65d
• JSPI WASM: 93ebb472969c4ccfe9d406e2ebd043da288a6d513a180a57827211278f4c82ed
• compat WASM: 91f1566e3dfa4cd591d902c03bfc2c92c9ceef8d5ce93cecae197ee1f4f5dd5d
• compat worker: 0b667db536815fb9e5dc9ffbfbd6ba29affdda38a4ba892d5372ddb41ce9a8c8
• JSPI Q1_0 profile: raw-layout vec4 WG64; Safari compat Q1_0 profile: scalar

The vendored runtime exposes the validated token-embedding WebGPU placement needed for single-graph Bonsai inference and fails closed when placement or the backend tripwire disagrees. Its exact JS/WASM hashes are checked during postinstall; source/patch provenance is in vendor/wllama-bonsai/SOURCE.json. Third-party notices are in THIRD_PARTY_NOTICES.md and public/licenses/.

The pinned WebGPU source implements Flash Attention with Q4_0/Q8_0 K/V. The benchmark lane has short 1.7B deterministic-output and memory evidence for both quantized types, a 27B long-state result only for Flash-off F16, and a 27B >8K smoke only for Flash-auto Q4_0. No 27B Q8_0 long-state result is claimed. Repeated and cross-device gates remain open, so Flash Attention remains off and F16/F16 remains the release default.

Licenses and attribution

The Bonsai GGUF tiers are Apache-2.0 model artifacts; their upstream notices are published in WaveCut/Bonsai-web-GGUF@v1.1.3/notices/. The shipped wllama/llama.cpp engines, React runtime, @noble/hashes, Shiki, and @shikijs/stream are MIT-licensed. Exact runtime license texts are included in public/licenses/; the consolidated inventory is THIRD_PARTY_NOTICES.md.