Instructions to use avinash2468/SmolVLM2-500M-GUI-Grounding with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use avinash2468/SmolVLM2-500M-GUI-Grounding with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="avinash2468/SmolVLM2-500M-GUI-Grounding") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("avinash2468/SmolVLM2-500M-GUI-Grounding") model = AutoModelForMultimodalLM.from_pretrained("avinash2468/SmolVLM2-500M-GUI-Grounding", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use avinash2468/SmolVLM2-500M-GUI-Grounding with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "avinash2468/SmolVLM2-500M-GUI-Grounding" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "avinash2468/SmolVLM2-500M-GUI-Grounding", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/avinash2468/SmolVLM2-500M-GUI-Grounding
- SGLang
How to use avinash2468/SmolVLM2-500M-GUI-Grounding with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "avinash2468/SmolVLM2-500M-GUI-Grounding" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "avinash2468/SmolVLM2-500M-GUI-Grounding", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "avinash2468/SmolVLM2-500M-GUI-Grounding" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "avinash2468/SmolVLM2-500M-GUI-Grounding", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use avinash2468/SmolVLM2-500M-GUI-Grounding with Docker Model Runner:
docker model run hf.co/avinash2468/SmolVLM2-500M-GUI-Grounding
SmolVLM2-500M-GUI-Grounding
A 500M-parameter vision-language model fine-tuned for GUI grounding: given a screenshot and a natural-language instruction ("click the export button"), it returns the point to click.
It is a fine-tune of SmolVLM2-500M-Video-Instruct, about 1 GB of weights.
It uses the same training recipe as the smaller avinash2468/SmolVLM-256M-GUI-Grounding; only the base model differs.
Coming soon: training and evaluation code, data details, and a technical report. Follow avinash2468 on Hugging Face or get in touch to hear when they are out.
Results
Accuracy (%) โ a prediction counts as correct when the predicted point lies inside the target element's box.
| inference | ScreenSpot-v2 (n = 1,272) | ScreenSpot-Pro (n = 1,581) | OSWorld-G (n = 510) |
|---|---|---|---|
| this model, 1 pass | 80.0 | 21.7 | 39.4 |
| this model, 4 passes | 74.8 | 40.5 | 46.5 |
| this model, 12 passes | 76.6 | 43.9 | 49.2 |
| 256M sibling, 1 pass | 74.8 | 14.3 | 33.3 |
| 256M sibling, 12 passes | 77.8 | 40.0 | 44.9 |
- 1 pass is a single forward pass on the full screenshot โ exactly what the quick start below does.
- 4 / 12 passes use a test-time refinement procedure that re-examines the screen before answering. It is not included in this release; those rows are reported for reference.
- Pick the setting per screen type. On high-resolution professional screens extra passes help a lot (ScreenSpot-Pro: 21.7 โ 43.9), but on ScreenSpot-v2 they lower accuracy (80.0 โ 76.6). For phone, web and ordinary desktop screens, a single pass is the better operating point.
- Compared with the 256M sibling, the larger language model helps in a single pass on every benchmark (+5.2 / +7.4 / +6.1).
Examples
Hand-picked single-pass predictions from this model on benchmark screenshots, chosen to show instructions that need some reading or reasoning. They are illustrations, not a random sample โ the table above is the real measure. In each image the red dot is the model's click, the green box is the ground-truth element, and the right panel zooms into the yellow area.
1. โClick the B21 cellโ โ Spreadsheet (OSWorld-G): the model has to work out column B and row 21 itself; the cell holds a name, not the text โB21โ.
2. โThree-dot menu icon to the right of 'Taylor Swift - Shake It Off' entryโ โ Browser history (OSWorld-G): one tiny icon out of a dozen identical ones, picked out by the row it belongs to.
3. โTurn on the alarm for 3:15โ โ Phone clock app (ScreenSpot-v2): the toggle on the 03:15 row, not one of the other alarms.
4. โswitch map layer to the opnvkarteโ โ OpenStreetMap (ScreenSpot-v2): matches a loosely spelled instruction to the โรPNVKarteโ layer.
5. โClick on the question icon of the "save" dialog.โ โ LibreOffice (OSWorld-G): a small icon inside a dialog stacked on top of two other windows.
Screenshots are from the ScreenSpot-v2 and OSWorld-G benchmarks.
Model details
| base model | HuggingFaceTB/SmolVLM2-500M-Video-Instruct (SigLIP-B/16 vision encoder + SmolLM2-360M language model) |
| parameters | 498.6M |
| weights | bf16 safetensors, โ 1.0 GB |
| training | full-parameter supervised fine-tuning only, in several stages, on public GUI and pointing data |
| training compute | โ 35 GPU-hours on one NVIDIA H100 80 GB |
| output | (x, y) with integers 0โ100, as a percentage of image width and height |
| language | English instructions |
Quick start
Requires transformers >= 5.0, torch, pillow, num2words (needed by the SmolVLM2 processor).
import math, re, torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
repo = "avinash2468/SmolVLM2-500M-GUI-Grounding"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval()
SYSTEM = ("You are a GUI grounding agent. You are given a screenshot and an instruction describing a single UI "
"element to interact with. Identify the location of that element.")
PROMPT = ("Instruction: {instruction}\n\n"
"Respond with ONLY the click location as coordinates in the format (x, y), where x and y are integers "
"from 0 to 100 giving the horizontal and vertical position as a fraction of the image width and height "
"times 100. Do not output anything else.")
def prepare(img):
"""Resize so the long side is a multiple of 512 px, clamped to 2048-2560 (the resolution used in training)."""
w, h = img.size
target = min(max(512 * math.ceil(max(w, h) / 512), 2048), 2560)
s = target / max(w, h)
return img.resize((round(w * s), round(h * s)), Image.LANCZOS)
def ground(image_path, instruction):
image = Image.open(image_path).convert("RGB")
view = prepare(image)
messages = [
{"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
{"role": "user", "content": [{"type": "image"},
{"type": "text", "text": PROMPT.format(instruction=instruction)}]},
]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[text], images=[[view]], return_tensors="pt",
images_kwargs={"size": {"longest_edge": max(view.size)}})
inputs = inputs.to(device, dtype=torch.bfloat16)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
answer = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
pairs = re.findall(r"(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)", answer)
if not pairs:
return None
x, y = (float(v) for v in pairs[-1])
return x / 100 * image.width, y / 100 * image.height # pixel coordinates in the original screenshot
print(ground("screenshot.png", "Click the Save button"))
Evaluation protocol
- Benchmarks: ScreenSpot-v2 (mobile, web, desktop; 1,272 items), ScreenSpot-Pro (high-resolution professional desktop apps; 1,581 items), OSWorld-G (Linux desktop; the 510 items with a ground-truth box โ the remaining refusal items are excluded).
- Metric: predicted point inside the ground-truth box; an unparseable answer counts as a miss.
- Decoding: greedy, batch size 32, bf16. Greedy outputs vary slightly with batch size and padding, so expect small (โฒ 0.5 point) differences when re-running.
- Checkpoint: the final checkpoint of training; no checkpoint selection on the test sets. Single training seed; no error bars.
- Contamination: exact-duplicate image checks found no overlap between the training data and the three benchmarks.
Limitations
- Icons are much harder than text. Most remaining errors are choosing the wrong one among visually similar adjacent icons.
- Very wide or very high-resolution screens (ultra-wide monitors, small targets on 4K displays) remain hard in a single pass.
- Not state of the art. Larger (2โ3B+) grounding models score substantially higher; the point of this model is its size.
- Grounding only. It predicts a single click point for a single-element instruction; it does not plan multi-step tasks, and it is not trained to refuse when the element is absent.
- The base model is the Video variant of SmolVLM2-500M (the only SmolVLM release at this size); this model is trained and evaluated on still screenshots only.
- Latency and memory on CPU / mobile hardware have not been measured yet.
- As with any GUI agent component, do not let it act on real systems without human oversight.
License
Apache-2.0, the same as the base model. The training data are publicly released datasets, each under its own license; please check those licenses for your use case.
Contact
Built by Avinash Prabhu Puppala. I'm open to research collaboration, compute partnerships and research roles on small / on-device GUI agents.
- Email: avinashprabhu2001@gmail.com
- LinkedIn: linkedin.com/in/avinash-prabhu-p
Citation
@misc{smolvlm2_500m_gui_grounding_2026,
title = {SmolVLM2-500M-GUI-Grounding: a 500M-parameter GUI grounding model},
author = {Puppala, Avinash Prabhu},
year = {2026},
howpublished = {\url{https://huggingface.co/avinash2468/SmolVLM2-500M-GUI-Grounding}}
}
- Downloads last month
- 39
Model tree for avinash2468/SmolVLM2-500M-GUI-Grounding
Base model
HuggingFaceTB/SmolLM2-360MCollection including avinash2468/SmolVLM2-500M-GUI-Grounding
Evaluation results
- Accuracy (1 pass) on ScreenSpot-v2test set self-reported80.000
- Accuracy (12 passes) on ScreenSpot-v2test set self-reported76.600
- Accuracy (1 pass) on ScreenSpot-Protest set self-reported21.700
- Accuracy (12 passes) on ScreenSpot-Protest set self-reported43.900
- Accuracy (1 pass) on OSWorld-G (510)test set self-reported39.400
- Accuracy (12 passes) on OSWorld-G (510)test set self-reported49.200




