laya-browser — laya fine-tuned as a browser-agent decision head (drop-in replacement for TypeSafe Jev)

46-second demo (mp4): v10s drives headless Chromium through jev-ultrafast, one encoder pass per step; recorded with code/apps/make_demo.py.

laya (convaiinnovations/laya) is a non-autoregressive "System 1" decision model: one bidirectional encoder pass answers several typed questions (choice / score / noul) with calibrated probabilities, no text generation. Out of the box it is near chance at browser decisions ("which element should I click for this goal?" — top-1 0.10 among ~45 candidates).

This repo is what it took to turn it into a usable decision head for browser-use/jev-ultrafast, whose /v1/systemone request format is identical to laya's predict(state, questions). Everything was done locally on one RTX 4070 Ti SUPER (16 GB), no paid API: the text helper and the DAgger teacher are a local Qwen3-8B-AWQ served by sglang.

What changed relative to the original laya

	original laya (typed-decisions)	this repo
browser decision quality (16 real tasks × 3 runs, corrected checks)	0 %	v10s: 62 %, v11s: 56 %, v10: 50 % (10 tasks pass 3/3, 6 fail 3/3; see below)
element top-1 on held-out pages (2,734 decisions, ~45 candidates)	0.10	0.66 (v10), 0.63 (v10s), 0.62 (v11s)
operation accuracy (CLICK / TYPE_TEXT / SELECT / DONE)	0.54	0.88–0.89
latency per browser step (3 questions, 30–65 candidates)	50–200 ms	41–50 ms (v10), 17–23 ms (v10s / v11s)
backbone	ModernBERT-large 421M	v10: same; v10s / v11s: mmBERT-base 322M
input format	jev's state verbatim (element table as JSON inside the state, truncated by the 1024-token window)	format v2/v3: elements live only in the option list (full label + role + current value), state keeps title / URL / history / 1.2–1.5k chars of text, head_max_len 512 → 768
training data	LocalLLaMA/typed-decisions	5,244 reverse-generated goals on 421 crawled pages (Qwen writes "the goal a user would state to need this element"), 700 real DONE states (clicks actually executed), 659 step-2 negatives, Mind2Web train (7,296 steps, candidates re-rendered as an element table), 177 on-policy DAgger corrections
training	—	laya's RLCD recipe (noisy-logit policy gradient + soft CE), single GPU, no gradient checkpointing, 4 epochs (~2 h for v10, ~1 h for v10s), post-hoc temperature
inference	HF eager + autocast	optional TileLang fast path (PR #25 to laya): fused GEMM/GEGLU/LayerNorm/RoPE, sliding-window flash attention, bf16-resident weights, CUDA graphs — 4–5× lower per-call latency, identical answers

Things that did not work (so you don't repeat them)

• Templated DONE goals ("Open the page titled X, stop once it is open") leak phrasing: the model learns stop when ⇒ DONE. DONE samples must be real landing pages after an executed action.
• If every DONE sample has exactly one prior action and every click sample has none, the model learns any history ⇒ DONE. Add mid-task negatives (step-2 goals on landing pages).
• Mind2Web alone kills DONE / TYPE_TEXT (no DONE there, CLICK dominates): re-weight rare operations (DONE ×4, TYPE_TEXT/SELECT ×3).
• Cutting page text to 3,000 chars saved nothing (the sequence is dominated by the head) and cost 0.04 top-1.
• torch.compile on variable-length batches recompiles per shape: 6× slower.
• Confidence-gated escalation to Qwen3-8B (System 2) made things worse (58 % → 42 %): on these pages the fine-tuned 322M/421M model is a better decider than an 8B general LLM. Use a stronger System 2 or none.
• jev's DOM reader hides password fields by design (login tasks are impossible) and never sees collapsed menus (Wikipedia's "Random article").

What still fails

The suite is bimodal: 10 tasks pass 3/3 (category / tab / page navigation, checkbox, <select>, HN pages, DuckDuckGo search in some runs) and 6 fail 3/3: "type then submit / pick a suggestion" flows (Wikipedia search ×2, arXiv), pagination that needs a scroll first (the model clicks the first visible item instead), and Google Flights. v11s added 682 scripted scroll / search-submit / select trajectories (code/finetune/rollouts.py): SELECT and DuckDuckGo improved, the scroll case did not — run-to-run variance on live sites (HN front page changes, DDG 50x pages) is larger than the v10s ↔ v11s difference, so treat the two as equivalent.

Teachers vs the fine-tuned student (80 held-out decisions, same element tables)

decider	op acc	target top-1	s / decision
Ternary-Bonsai-2-27B (local, thinking off)	0.825	0.554	1.56
Bonsai-27B, thinking budget 300 tokens	0.861	0.603	4.7
Bonsai-27B, thinking unrestricted	0.625	0.474	11
Qwen3-8B-AWQ (35 % of requests failed, survivors only)	0.904	0.617	0.38
laya v11s (322M, this repo; full 2,734-case set)	0.890	0.623	0.021

A 27B general model with a short thinking budget matches the 322M fine-tuned student at 200× the latency; neither 8B nor 27B is a useful DAgger teacher or System-2 fallback here. Further gains need a stronger teacher or more targeted trajectories.

Files

v10/     ModernBERT-large 421M, format v2, head_max_len 768   (best held-out top-1)
v10s/    mmBERT-base 322M,      format v3, head_max_len 768   (17–23 ms per step; best on the live suite)
v11s/    v10s data + 682 scripted scroll / search-submit / select trajectories (equivalent to v10s within noise)
code/    finetune pipeline, laya systemone server, task suite, TileLang kernels, jev-ultrafast patch
results/ per-run suite JSONs and logs behind every number above

Each checkpoint is a laya checkpoint directory (model.safetensors, encoder/, tokenizer/, rl_agent_config.json); the config records laya_fmt and head_max_len_train so the server applies the matching input format automatically.

Use

huggingface-cli download cklxx/laya-browser --local-dir laya-browser
cd laya-browser/code && uv sync --extra fast     # pinned uv.lock (Python 3.12, torch 2.11, tilelang 0.1.14)
uv run python verify.py v10s                     # downloads v10s if needed, answers one recorded browser step
uv run python verify.py v10s --fast              # same through the TileLang fast path

Verified from a clean environment on 2026-09-21 (RTX 4070 Ti SUPER): TYPE_TEXT → [2] Search Wikipedia (searchbox), 35 ms per step stock / 28 ms with the fast path on a 65-option, 2.5k-token step. Extras: --extra data (Mind2Web conversion, dataset eval), --extra browser (live suite / crawling; also needs jev-ultrafast with code/jev-ultrafast.patch applied and a Chromium with --remote-debugging-port=9222).

import laya
agent = laya.load("laya-browser/v10s")                 # a local laya checkpoint dir
agent.cfg["head_max_len"] = agent.cfg["head_max_len_train"]
# state / questions exactly as jev-ultrafast's model.choose() builds them, after the format-v3 transform in code/apps/systemone_server.py
result = agent.predict(state, questions)

As a TypeSafe replacement for jev-ultrafast:

# in code/: laya systemone-compatible server (format transform + optional gating + DAgger logging)
python apps/systemone_server.py 8791 /path/to/laya-browser/v10s 999
# in jev-ultrafast (apply code/jev-ultrafast.patch): TYPESAFE_BASE_URL=http://127.0.0.1:8791

code/apps/browser_suite.py runs the 16-task real-browser suite with automatic outcome checks (REPEATS=3).

Reproduce

code/finetune/README.md documents every step (crawl → reverse-generate goals → execute clicks for DONE → step-2 → Mind2Web conversion → DAgger → build → train → calibrate → eval → suite) with the exact scripts (run_v10.sh, run_v10s.sh, run_final.sh) and all intermediate numbers from v1 to v10s.

GPU cost

v10s: ~0.65 GB weights, ~1.5 GB VRAM resident with CUDA graphs, 17–23 ms per 3-question browser step, 3 ms for a single-question call. v10: ~0.85 GB weights, ~1.8 GB VRAM, 41–50 ms per step. The Qwen text helper (only needed for TYPE_TEXT values) is separate.

License

Apache-2.0, same as laya. Mind2Web is used under its own license for training only.