--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: image-text-to-text base_model: - HuggingFaceTB/SmolVLM-256M-Instruct tags: - gui-grounding - gui-agent - computer-use - ui-grounding - screen-understanding - smolvlm model-index: - name: SmolVLM-256M-GUI-Grounding results: - task: type: image-text-to-text name: GUI grounding dataset: name: ScreenSpot-v2 type: screenspot-v2 split: test metrics: - type: accuracy value: 74.8 name: Accuracy (1 pass) - type: accuracy value: 77.8 name: Accuracy (12 passes) - task: type: image-text-to-text name: GUI grounding dataset: name: ScreenSpot-Pro type: screenspot-pro split: test metrics: - type: accuracy value: 14.3 name: Accuracy (1 pass) - type: accuracy value: 40.0 name: Accuracy (12 passes) - task: type: image-text-to-text name: GUI grounding dataset: name: OSWorld-G (510) type: osworld-g split: test metrics: - type: accuracy value: 33.3 name: Accuracy (1 pass) - type: accuracy value: 44.9 name: Accuracy (12 passes) --- # SmolVLM-256M-GUI-Grounding A **256M-parameter** vision-language model fine-tuned for **GUI grounding**: given a screenshot and a natural-language instruction ("click the export button"), it returns the point to click. It is a fine-tune of [SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct), about 0.5 GB of weights, small enough for laptops and on-device use. A larger sibling trained with the same recipe is [avinash2468/SmolVLM2-500M-GUI-Grounding](https://huggingface.co/avinash2468/SmolVLM2-500M-GUI-Grounding). > **Coming soon:** training and evaluation code, data details, and a technical report. Follow [avinash2468](https://huggingface.co/avinash2468) on Hugging Face or [get in touch](#contact) to hear when they are out. ## Results Accuracy (%) — a prediction counts as correct when the predicted point lies inside the target element's box. | inference | ScreenSpot-v2 (n = 1,272) | ScreenSpot-Pro (n = 1,581) | OSWorld-G (n = 510) | |---|---:|---:|---:| | base SmolVLM-256M-Instruct, zero-shot | 2.1 | ~0 | ~0 | | **this model, 1 pass** | **74.8** | **14.3** | **33.3** | | this model, 4 passes | 76.0 | 37.3 | 44.7 | | **this model, 12 passes** | **77.8** | **40.0** | **44.9** | - **1 pass** is a single forward pass on the full screenshot — exactly what the [quick start](#quick-start) below does. - **4 / 12 passes** use a test-time refinement procedure that re-examines the screen before answering. It is **not included in this release**; those rows are reported for reference. - Extra passes matter most on high-resolution professional screens (ScreenSpot-Pro: 14.3 → 40.0) and little on ScreenSpot-v2, where a single pass already reads the screen well. - Twelve passes of a 256M model cost roughly as much compute as one pass of a 3B model, so the 12-pass numbers should be compared with ~3B models at similar compute, not read as a free efficiency gain. ## Examples Hand-picked single-pass predictions from this model on benchmark screenshots, chosen to show instructions that need some reading or reasoning. They are illustrations, not a random sample — the table above is the real measure. In each image the **red dot** is the model's click, the **green box** is the ground-truth element, and the right panel zooms into the **yellow** area. **1. “Click the B21 cell”** — Spreadsheet (OSWorld-G): the model has to work out column B and row 21 itself; the cell holds a name, not the text “B21”. ![Click the B21 cell](assets/example_1.jpg) **2. “Three-dot menu icon to the right of 'Taylor Swift - Shake It Off' entry”** — Browser history (OSWorld-G): one tiny icon out of a dozen identical ones, picked out by the row it belongs to. ![Three-dot menu icon to the right of 'Taylor Swift - Shake It Off' entry](assets/example_2.jpg) **3. “Turn on the alarm for 3:15”** — Phone clock app (ScreenSpot-v2): the toggle on the 03:15 row, not one of the other alarms. ![Turn on the alarm for 3:15](assets/example_3.jpg) **4. “switch map layer to the opnvkarte”** — OpenStreetMap (ScreenSpot-v2): matches a loosely spelled instruction to the “ÖPNVKarte” layer. ![switch map layer to the opnvkarte](assets/example_4.jpg) **5. “Click on the question icon of the "save" dialog.”** — LibreOffice (OSWorld-G): a small icon inside a dialog stacked on top of two other windows. ![Click on the question icon of the "save" dialog.](assets/example_5.jpg) Screenshots are from the ScreenSpot-v2 and OSWorld-G benchmarks. ## Model details | | | |---|---| | base model | HuggingFaceTB/SmolVLM-256M-Instruct (SigLIP-B/16 vision encoder + SmolLM2-135M language model) | | parameters | 251.2M | | weights | bf16 safetensors, ≈ 0.5 GB | | training | full-parameter supervised fine-tuning only, in several stages, on public GUI and pointing data | | training compute | ≈ 30 GPU-hours on one NVIDIA H100 80 GB | | output | `(x, y)` with integers 0–100, as a percentage of image width and height | | language | English instructions | ## Quick start Requires `transformers >= 5.0`, `torch`, `pillow`. ```python import math, re, torch from PIL import Image from transformers import AutoProcessor, AutoModelForImageTextToText repo = "avinash2468/SmolVLM-256M-GUI-Grounding" device = "cuda" if torch.cuda.is_available() else "cpu" processor = AutoProcessor.from_pretrained(repo) model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval() SYSTEM = ("You are a GUI grounding agent. You are given a screenshot and an instruction describing a single UI " "element to interact with. Identify the location of that element.") PROMPT = ("Instruction: {instruction}\n\n" "Respond with ONLY the click location as coordinates in the format (x, y), where x and y are integers " "from 0 to 100 giving the horizontal and vertical position as a fraction of the image width and height " "times 100. Do not output anything else.") def prepare(img): """Resize so the long side is a multiple of 512 px, clamped to 2048-2560 (the resolution used in training).""" w, h = img.size target = min(max(512 * math.ceil(max(w, h) / 512), 2048), 2560) s = target / max(w, h) return img.resize((round(w * s), round(h * s)), Image.LANCZOS) def ground(image_path, instruction): image = Image.open(image_path).convert("RGB") view = prepare(image) messages = [ {"role": "system", "content": [{"type": "text", "text": SYSTEM}]}, {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": PROMPT.format(instruction=instruction)}]}, ] text = processor.apply_chat_template(messages, add_generation_prompt=True) inputs = processor(text=[text], images=[[view]], return_tensors="pt", images_kwargs={"size": {"longest_edge": max(view.size)}}) inputs = inputs.to(device, dtype=torch.bfloat16) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=40, do_sample=False) answer = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0] pairs = re.findall(r"(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)", answer) if not pairs: return None x, y = (float(v) for v in pairs[-1]) return x / 100 * image.width, y / 100 * image.height # pixel coordinates in the original screenshot print(ground("screenshot.png", "Click the Save button")) ``` ## Evaluation protocol - **Benchmarks:** ScreenSpot-v2 (mobile, web, desktop; 1,272 items), ScreenSpot-Pro (high-resolution professional desktop apps; 1,581 items), OSWorld-G (Linux desktop; the 510 items with a ground-truth box — the remaining refusal items are excluded). - **Metric:** predicted point inside the ground-truth box; an unparseable answer counts as a miss. - **Decoding:** greedy, batch size 32, bf16. Greedy outputs vary slightly with batch size and padding, so expect small (≲ 0.5 point) differences when re-running. - **Checkpoint:** the final checkpoint of training; no checkpoint selection on the test sets. Single training seed; no error bars. - **Contamination:** exact-duplicate image checks found no overlap between the training data and the three benchmarks. ## Limitations - **Icons are much harder than text.** Most remaining errors are choosing the wrong one among visually similar adjacent icons. - **Very wide or very high-resolution screens** (ultra-wide monitors, small targets on 4K displays) remain hard in a single pass. - **Not state of the art.** Larger (2–3B+) grounding models score substantially higher; the point of this model is its size. - **Grounding only.** It predicts a single click point for a single-element instruction; it does not plan multi-step tasks, and it is not trained to refuse when the element is absent. - Latency and memory on CPU / mobile hardware have not been measured yet. - As with any GUI agent component, do not let it act on real systems without human oversight. ## License Apache-2.0, the same as the base model. The training data are publicly released datasets, each under its own license; please check those licenses for your use case. ## Contact Built by **Avinash Prabhu Puppala**. I'm open to research collaboration, compute partnerships and research roles on small / on-device GUI agents. - Email: **avinashprabhu2001@gmail.com** - LinkedIn: [linkedin.com/in/avinash-prabhu-p](https://www.linkedin.com/in/avinash-prabhu-p/) ## Citation ```bibtex @misc{smolvlm256m_gui_grounding_2026, title = {SmolVLM-256M-GUI-Grounding: a 256M-parameter GUI grounding model}, author = {Puppala, Avinash Prabhu}, year = {2026}, howpublished = {\url{https://huggingface.co/avinash2468/SmolVLM-256M-GUI-Grounding}} } ```