how.md

How it works

The assistant turn is a fixed template; each slot ends in a placeholder:

action: ___ x: ___ y: ___ x2: ___ y2: ___ key: ___ sdir: ___ samt: ___

A slot's answer is the next-token distribution at the token before its placeholder, restricted to that slot's labels: the 11 action words for action, and 256 bins for each coordinate, every bin a single byte-like token. One forward pass over the whole template reads every slot at once. A LoRA trained with cross-entropy against soft (Gaussian) targets puts the probability on the right bins, so the distribution is meaningful: the heatmap is P(x) × P(y).

What to type, or a final answer, is free text: for those steps the base model (adapter switched off) writes it.

Boxes work the same way: centre x, centre y, and a log-scaled width and height, four slots.

Measured (blog post, Gemma 4 31B NVFP4 on vLLM, one RTX PRO 6000)

how the step is decided	time per step
slot read	146 ms
vanilla, native [y, x] JSON (~20 tokens)	472–546 ms
vanilla + thinking (130–230 tokens)	2.6–3.1 s

this model (E3)
ScreenSpot clicks (1,272)	0.866
computer-use steps, held-out AGUVIS (1,600)	0.659
RefCOCO val photo boxes, mean IoU (3,811)	0.770
COCO val2017 boxes, mean IoU (1,000)	0.742

Limits

Trained on web, desktop and mobile screenshots (AGUVIS, Wave-UI) and RefCOCO photos. It decides one step at a time from the screenshot, the instruction and the previous actions you give it; it does not act on anything. Expect mistakes on small targets and on interfaces unlike its training data.