--- license: mit base_model: mPLUG/GUI-Owl-1.5-2B-Instruct library_name: transformers pipeline_tag: image-text-to-text tags: [computer-use, gui-agent, gui-grounding, slot-reading] --- # GUI-Decisions-2B [![Slot read vs writing JSON: the same model on the same Wikipedia task, 4 decisions in 0.63 s against 2.08 s](assets/hero.gif)](https://huggingface.co/spaces/infinitylogesh/GUI-Decisions) *Slot reading with the 31B model (from the blog post): reading slots (left, 158 ms per decision) against writing the step as JSON (right, 520 ms). [Try the 31B model in the Space](https://huggingface.co/spaces/infinitylogesh/GUI-Decisions).* [mPLUG/GUI-Owl-1.5-2B-Instruct](https://huggingface.co/mPLUG/GUI-Owl-1.5-2B-Instruct) with the slot-reading LoRA merged in (bf16), which makes a computer-use step **readable from one forward pass** instead of decoded: the action, the point (256 bins per axis), a swipe's end, the key and the scroll, as next-token distributions at fixed slots. The small-model result of the blog post [Stop Decoding Coordinates: 3× Faster Computer-Use Grounding](https://logeshumapathi.com/blog/2026/10/06/slot-reading.html). The LoRA alone: [GUI-Decisions-2B-LoRA](https://huggingface.co/infinitylogesh/GUI-Decisions-2B-LoRA); the 31B version: [GUI-Decisions-31B](https://huggingface.co/infinitylogesh/GUI-Decisions-31B). ## Results Accuracy (blog post): | | computer-use step (held-out AGUVIS, 1,600) | ScreenSpot clicks (1,272) | | --- | --- | --- | | **GUI-Owl-2B + the LoRA** | **0.626** | **0.770** | | GUI-Owl-2B (its own tool-call format), native resolution | 0.626 | 0.645 | | GUI-Owl-2B, same 1 MP images | 0.610 | 0.626 | Vanilla GUI-Owl's click numbers use its agent prompt, not its dedicated grounding prompt. This merged model, read in-process with transformers (bf16): ScreenSpot clicks 0.763 against the published LoRA's 0.770 on the same cases (paired difference −0.006 [−0.013, +0.001], not significant; the unmerged LoRA read the same way scores 0.764), and computer-use steps 0.627 on the held-out AGUVIS set. Latency of one computer-use step, measured with the LoRA on the base model (this merged model reads at the same speed: ~83 against ~87 ms end to end): vLLM 0.30, one RTX PRO 6000, cold screenshots, one request at a time, all conditions in one session (`scripts/gui_owl.py bench 80`): | how the step is decided | median (p90) | | --- | --- | | **slots, lazy**: `action`, `x`, `y`, then only the slots the action needs (what `decide` does) | **65 ms** (71) | | slots, the whole template (8 slots) | 93 ms (104) | | GUI-Owl-2B writing its tool call (~50 tokens), same 1 MP images | 194 ms (217) | | GUI-Owl-2B writing its tool call (~50 tokens), native resolution | 323 ms (391) | The blog's earlier session measured 133, 225 and 444 ms for the last three rows on another box; the ratios agree (whole-template slots at ~30% of native-resolution GUI-Owl). End to end through `decide`, including the client's JPEG encoding, a step takes ~85 ms. ## Use ### With vLLM (recommended) Served on one RTX PRO 6000 a computer-use step takes ~83 ms end to end through the code below (median over 60 cold screenshots, client JPEG encoding included; the LoRA on the base model: ~87 ms). **1. Install** vLLM 0.30 and the reader (`slotread`, from the code repository): ```bash git clone https://github.com/infinitylogesh/gui-decisions && cd gui-decisions pip install "vllm==0.30.*" -e . bash serving/patch_vllm.sh # recommended: 256 label ids per request ``` **2. Serve:** ```bash vllm serve infinitylogesh/GUI-Decisions-2B --served-model-name gui-decisions-2b --port 8001 \ --max-model-len 16384 --gpu-memory-utilization 0.6 --max-logprobs 300 --enable-prefix-caching \ --mm-processor-cache-gb 0 --api-server-count 8 --limit-mm-per-prompt '{"image":1}' ``` (`MODEL=infinitylogesh/GUI-Decisions-2B NAME=gui-decisions-2b MAX_LEN=16384 GPU_UTIL=0.6 EXTRA_ARGS='--limit-mm-per-prompt {"image":1}' bash serving/serve_vllm.sh` runs the same command.) **3. Read** steps: ```python from PIL import Image from slotread.client import SlotModel m = SlotModel("gui-decisions-2b", "infinitylogesh/GUI-Decisions-2B") # server: VLLM_URL, default http://127.0.0.1:8001 step = m.decide(Image.open("screenshot.png"), "open the settings") # {'action': 'click', 'x': ..., 'y': ..., 'ms': ...} ``` `decide` reads `action`, `x` and `y` first, then only the slots the action needs. Screenshots are downscaled to at most 1,048,576 pixels, as in training (`image_cap` in `slotread_config.json`; `SlotModel` applies it). **What a read is.** For each slot, `SlotModel` sends one chat request whose assistant turn is prefilled up to that slot, and asks vLLM for the log-probabilities of exactly that slot's labels; nothing is generated beyond one token. The requests for one step run in parallel and share the image through vLLM's prefix cache: ```json {"model": "gui-decisions-2b", "messages": [{"role": "system", "content": [{"type": "text", "text": ""}]}, {"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}, {"type": "text", "text": "Please generate the next move according to the UI screenshot, instruction and previous actions.\n\nInstruction: turn on dark mode\n\nPrevious actions:\nNone"}]}, {"role": "assistant", "content": [{"type": "text", "text": "action: ___ x:"}]}], "add_generation_prompt": false, "continue_final_message": true, "chat_template_kwargs": {"enable_thinking": false}, "max_tokens": 1, "temperature": 0, "logprobs": true, "top_logprobs": 1, "return_tokens_as_token_ids": true, "logprob_token_ids": [""]} ``` The slot's answer is the softmax over those labels: the argmax for `action`; for a coordinate, the argmax bin refined by its ±2 neighbours (bin *k* of 256 is the value (*k* + 0.5) / 256 across the screen). Notes: - `patch_vllm.sh` raises vLLM's per-request `logprob_token_ids` cap from 128 to 256, so a 256-bin slot is one request. Without it, set `MAX_LABEL_IDS=128` for the client (each coordinate then takes two requests). - `--api-server-count 8`: a single API-server process serializes a step's parallel slot requests. `--mm-processor-cache-gb 0`: vLLM 0.30's media cache raced on concurrent requests for the same image. - `previous` takes the earlier steps as numbered lines, e.g. `"1. click(One way)\n2. click(From)"`. ### In-process with transformers (no server) ```python from PIL import Image from slotread.local import LocalSlotModel m = LocalSlotModel.from_pretrained("infinitylogesh/GUI-Decisions-2B") print(m.decide(Image.open("screenshot.png"), "open the settings")) ``` ## Training LoRA r 16, alpha 32, on the language model's attention and MLP projections; AGUVIS computer-use steps (5,000) and Wave-UI element clicks (2,000); one RTX PRO 6000, about 40 minutes. ## Limits One step at a time, no memory and no acting; computer use only (no boxes or masks). Text arguments (what to type, the final answer) need another writer: asked with our plain prompt, base GUI-Owl answered a city to type with the field's placeholder text. Trained on AGUVIS and Wave-UI; check their terms for your use. ## Citation ```bibtex @article{umapathi2026slotreading, title = "Stop Decoding Coordinates: 3.5× Faster Computer-Use Grounding", author = "Umapathi, Logesh Kumar", journal = "logeshumapathi.com", year = "2026", month = "Oct", url = "https://logeshumapathi.com/blog/2026/10/06/slot-reading.html" } ```