SmolVLM-256M-GUI-Grounding

A 256M-parameter vision-language model fine-tuned for GUI grounding: given a screenshot and a natural-language instruction ("click the export button"), it returns the point to click.

It is a fine-tune of SmolVLM-256M-Instruct, about 0.5 GB of weights, small enough for laptops and on-device use.

A larger sibling trained with the same recipe is avinash2468/SmolVLM2-500M-GUI-Grounding.

Coming soon: training and evaluation code, data details, and a technical report. Follow avinash2468 on Hugging Face or get in touch to hear when they are out.

Results

Accuracy (%) โ€” a prediction counts as correct when the predicted point lies inside the target element's box.

inference ScreenSpot-v2 (n = 1,272) ScreenSpot-Pro (n = 1,581) OSWorld-G (n = 510)
base SmolVLM-256M-Instruct, zero-shot 2.1 ~0 ~0
this model, 1 pass 74.8 14.3 33.3
this model, 4 passes 76.0 37.3 44.7
this model, 12 passes 77.8 40.0 44.9
  • 1 pass is a single forward pass on the full screenshot โ€” exactly what the quick start below does.
  • 4 / 12 passes use a test-time refinement procedure that re-examines the screen before answering. It is not included in this release; those rows are reported for reference.
  • Extra passes matter most on high-resolution professional screens (ScreenSpot-Pro: 14.3 โ†’ 40.0) and little on ScreenSpot-v2, where a single pass already reads the screen well.
  • Twelve passes of a 256M model cost roughly as much compute as one pass of a 3B model, so the 12-pass numbers should be compared with ~3B models at similar compute, not read as a free efficiency gain.

Examples

Hand-picked single-pass predictions from this model on benchmark screenshots, chosen to show instructions that need some reading or reasoning. They are illustrations, not a random sample โ€” the table above is the real measure. In each image the red dot is the model's click, the green box is the ground-truth element, and the right panel zooms into the yellow area.

1. โ€œClick the B21 cellโ€ โ€” Spreadsheet (OSWorld-G): the model has to work out column B and row 21 itself; the cell holds a name, not the text โ€œB21โ€.

Click the B21 cell

2. โ€œThree-dot menu icon to the right of 'Taylor Swift - Shake It Off' entryโ€ โ€” Browser history (OSWorld-G): one tiny icon out of a dozen identical ones, picked out by the row it belongs to.

Three-dot menu icon to the right of 'Taylor Swift - Shake It Off' entry

3. โ€œTurn on the alarm for 3:15โ€ โ€” Phone clock app (ScreenSpot-v2): the toggle on the 03:15 row, not one of the other alarms.

Turn on the alarm for 3:15

4. โ€œswitch map layer to the opnvkarteโ€ โ€” OpenStreetMap (ScreenSpot-v2): matches a loosely spelled instruction to the โ€œร–PNVKarteโ€ layer.

switch map layer to the opnvkarte

5. โ€œClick on the question icon of the "save" dialog.โ€ โ€” LibreOffice (OSWorld-G): a small icon inside a dialog stacked on top of two other windows.

Click on the question icon of the "save" dialog.

Screenshots are from the ScreenSpot-v2 and OSWorld-G benchmarks.

Model details

base model HuggingFaceTB/SmolVLM-256M-Instruct (SigLIP-B/16 vision encoder + SmolLM2-135M language model)
parameters 251.2M
weights bf16 safetensors, โ‰ˆ 0.5 GB
training full-parameter supervised fine-tuning only, in several stages, on public GUI and pointing data
training compute โ‰ˆ 30 GPU-hours on one NVIDIA H100 80 GB
output (x, y) with integers 0โ€“100, as a percentage of image width and height
language English instructions

Quick start

Requires transformers >= 5.0, torch, pillow.

import math, re, torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "avinash2468/SmolVLM-256M-GUI-Grounding"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval()

SYSTEM = ("You are a GUI grounding agent. You are given a screenshot and an instruction describing a single UI "
          "element to interact with. Identify the location of that element.")
PROMPT = ("Instruction: {instruction}\n\n"
          "Respond with ONLY the click location as coordinates in the format (x, y), where x and y are integers "
          "from 0 to 100 giving the horizontal and vertical position as a fraction of the image width and height "
          "times 100. Do not output anything else.")

def prepare(img):
    """Resize so the long side is a multiple of 512 px, clamped to 2048-2560 (the resolution used in training)."""
    w, h = img.size
    target = min(max(512 * math.ceil(max(w, h) / 512), 2048), 2560)
    s = target / max(w, h)
    return img.resize((round(w * s), round(h * s)), Image.LANCZOS)

def ground(image_path, instruction):
    image = Image.open(image_path).convert("RGB")
    view = prepare(image)
    messages = [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
        {"role": "user", "content": [{"type": "image"},
                                     {"type": "text", "text": PROMPT.format(instruction=instruction)}]},
    ]
    text = processor.apply_chat_template(messages, add_generation_prompt=True)
    inputs = processor(text=[text], images=[[view]], return_tensors="pt",
                       images_kwargs={"size": {"longest_edge": max(view.size)}})
    inputs = inputs.to(device, dtype=torch.bfloat16)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
    answer = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
    pairs = re.findall(r"(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)", answer)
    if not pairs:
        return None
    x, y = (float(v) for v in pairs[-1])
    return x / 100 * image.width, y / 100 * image.height  # pixel coordinates in the original screenshot

print(ground("screenshot.png", "Click the Save button"))

Evaluation protocol

  • Benchmarks: ScreenSpot-v2 (mobile, web, desktop; 1,272 items), ScreenSpot-Pro (high-resolution professional desktop apps; 1,581 items), OSWorld-G (Linux desktop; the 510 items with a ground-truth box โ€” the remaining refusal items are excluded).
  • Metric: predicted point inside the ground-truth box; an unparseable answer counts as a miss.
  • Decoding: greedy, batch size 32, bf16. Greedy outputs vary slightly with batch size and padding, so expect small (โ‰ฒ 0.5 point) differences when re-running.
  • Checkpoint: the final checkpoint of training; no checkpoint selection on the test sets. Single training seed; no error bars.
  • Contamination: exact-duplicate image checks found no overlap between the training data and the three benchmarks.

Limitations

  • Icons are much harder than text. Most remaining errors are choosing the wrong one among visually similar adjacent icons.
  • Very wide or very high-resolution screens (ultra-wide monitors, small targets on 4K displays) remain hard in a single pass.
  • Not state of the art. Larger (2โ€“3B+) grounding models score substantially higher; the point of this model is its size.
  • Grounding only. It predicts a single click point for a single-element instruction; it does not plan multi-step tasks, and it is not trained to refuse when the element is absent.
  • Latency and memory on CPU / mobile hardware have not been measured yet.
  • As with any GUI agent component, do not let it act on real systems without human oversight.

License

Apache-2.0, the same as the base model. The training data are publicly released datasets, each under its own license; please check those licenses for your use case.

Contact

Built by Avinash Prabhu Puppala. I'm open to research collaboration, compute partnerships and research roles on small / on-device GUI agents.

Citation

@misc{smolvlm256m_gui_grounding_2026,
  title        = {SmolVLM-256M-GUI-Grounding: a 256M-parameter GUI grounding model},
  author       = {Puppala, Avinash Prabhu},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/avinash2468/SmolVLM-256M-GUI-Grounding}}
}
Downloads last month
47
Safetensors
Model size
0.3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for avinash2468/SmolVLM-256M-GUI-Grounding

Collection including avinash2468/SmolVLM-256M-GUI-Grounding

Evaluation results