README.md

---
license: apache-2.0
datasets:
- docling-project/screenparse
tags:
- text-generation
- screen-parsing
- ui-understanding
- object-detection
- grounding
- web
- screentag
- docling
- granite
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
---

# ScreenVLM

**ScreenVLM** is a compact (316M-parameter) multimodal vision-language model for **complete screen parsing** — detecting, classifying, and localizing all UI elements on a web page screenshot. Given an image, it produces a structured **ScreenTag** representation with bounding boxes, semantic labels (55 UI element classes), and text content for every visible element.

- **Developed by**: IBM Research Zurich - ETH Zurich
- **Model type**: Multi-modal model (image+text-to-text)
- **Language(s)**: English
- **License**: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
- **Paper**: [ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing](TODO)
- **Code**: [GitHub](TODO)
- **Dataset**: [docling-project/screenparse](https://huggingface.co/docling-project/screenparse)

## Model Summary

ScreenVLM builds upon the [Idefics3](https://huggingface.co/docs/transformers/en/model_doc/idefics3) architecture with two key modifications: it uses [siglip2-base-patch16-512](https://huggingface.co/google/siglip2-base-patch16-512) as the vision encoder and a Granite 165M LLM as the language backbone. The model was trained on **ScreenParse**, a large-scale dataset of 771K web screenshots with dense UI element annotations across 55 semantic classes.

### Key Features

- **Complete screen parsing**: Detects all UI elements on a page, not just sparse grounding targets
- **55 UI element classes**: Buttons, links, inputs, navigation bars, menus, images, and more
- **ScreenTag output format**: Structured, hierarchical representation with bounding boxes and text
- **Compact size**: ~258M parameters (714MB safetensors), enabling fast inference

## Output Format

ScreenVLM generates output in **ScreenTag** format — a structured representation where each UI element is wrapped in semantic tags with location tokens:

```
<screentag>
<button><loc_10><loc_20><loc_50><loc_35>Submit</button>
<link><loc_100><loc_200><loc_180><loc_210>Learn more</link>
<navigation_bar><loc_0><loc_0><loc_500><loc_30>
  <link><loc_10><loc_5><loc_60><loc_25>Home</link>
  <link><loc_70><loc_5><loc_120><loc_25>About</link>
</navigation_bar>
</screentag>
```

Each `<loc_X>` token represents a coordinate in the normalized [0, 500] space. Four consecutive location tokens define `<left><top><right><bottom>` of the bounding box.

## Usage

### Inference with Transformers

```python
import re
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
MODEL_PATH = "docling-project/ScreenVLM"
NORM_SIZE = 500

# Load image
image = load_image("https://example.com/screenshot.png")

# Initialize processor and model
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = AutoModelForVision2Seq.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.bfloat16,
    _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "sdpa",
).to(DEVICE)

# Create input
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Generate the screen representation for this UI:"},
        ],
    },
]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to(DEVICE)

# Generate
generated_ids = model.generate(**inputs, max_new_tokens=6192)
prompt_length = inputs.input_ids.shape[1]
output = processor.batch_decode(
    generated_ids[:, prompt_length:],
    skip_special_tokens=False,
)[0].lstrip()

# Parse ScreenTag output into structured UI elements
def parse_screentag(text, width, height):
    pattern = re.compile(
        r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>"
        r"\s*<loc_(?P<l>\d+)><loc_(?P<t>\d+)><loc_(?P<r>\d+)><loc_(?P<b>\d+)>"
        r"(?P<text>[^<]*)"
    )
    elements = []
    for m in pattern.finditer(text):
        l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")]
        if r < l: l, r = r, l
        if b < t: t, b = b, t
        x = l / NORM_SIZE * width
        y = t / NORM_SIZE * height
        w = (r - l) / NORM_SIZE * width
        h = (b - t) / NORM_SIZE * height
        elements.append({
            "label": m.group("tag"),
            "bbox": (x, y, w, h),
            "text": m.group("text").strip() or None,
        })
    return elements

elements = parse_screentag(output, *image.size)
for el in elements:
    print(f"{el['label']:20s} bbox=({int(el['bbox'][0]):4d},{int(el['bbox'][1]):4d},{int(el['bbox'][2]):4d},{int(el['bbox'][3]):4d})  text={el['text']!r}")
```

### Batch Inference with vLLM

```python
import os
import re
import time
from vllm import LLM, SamplingParams
from transformers import AutoProcessor
from PIL import Image

MODEL_PATH = "docling-project/ScreenVLM"
IMAGE_DIR = "screenshots/"
PROMPT_TEXT = "Generate the screen representation for this UI:"
NORM_SIZE = 500

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": PROMPT_TEXT},
        ],
    },
]

# Initialize
llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})
processor = AutoProcessor.from_pretrained(MODEL_PATH)

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=6192,
    skip_special_tokens=False,
)

# Build batch
batched_inputs = []
image_sizes = []

for img_file in sorted(os.listdir(IMAGE_DIR)):
    if img_file.lower().endswith((".png", ".jpg", ".jpeg")):
        img_path = os.path.join(IMAGE_DIR, img_file)
        image = Image.open(img_path).convert("RGB")
        prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
        batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}})
        image_sizes.append((img_file, image.size))

# Run batch inference
start = time.time()
outputs = llm.generate(batched_inputs, sampling_params=sampling_params)

# Parse ScreenTag output into structured UI elements
def parse_screentag(text, width, height):
    pattern = re.compile(
        r"<(?P<tag>[a-zA-Z][a-zA-Z0-9_]*)>"
        r"\s*<loc_(?P<l>\d+)><loc_(?P<t>\d+)><loc_(?P<r>\d+)><loc_(?P<b>\d+)>"
        r"(?P<text>[^<]*)"
    )
    elements = []
    for m in pattern.finditer(text):
        l, t, r, b = [max(0, min(int(m.group(k)), NORM_SIZE)) for k in ("l", "t", "r", "b")]
        if r < l: l, r = r, l
        if b < t: t, b = b, t
        x = l / NORM_SIZE * width
        y = t / NORM_SIZE * height
        w = (r - l) / NORM_SIZE * width
        h = (b - t) / NORM_SIZE * height
        elements.append({
            "label": m.group("tag"),
            "bbox": (x, y, w, h),
            "text": m.group("text").strip() or None,
        })
    return elements

for output, (name, (w, h)) in zip(outputs, image_sizes):
    screentag = output.outputs[0].text
    elements = parse_screentag(screentag, w, h)
    print(f"--- {name} ({len(elements)} elements) ---")
    for el in elements:
        print(f"  {el['label']:20s} bbox=({int(el['bbox'][0]):4d},{int(el['bbox'][1]):4d},{int(el['bbox'][2]):4d},{int(el['bbox'][3]):4d})  text={el['text']!r}")

print(f"\nTotal: {time.time() - start:.1f}s for {len(batched_inputs)} images")
```

## Training

ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework with 16 NVIDIA H100 GPUs.

**Training data**: [ScreenParse](https://huggingface.co/docling-project/screenparse) — 771K web page screenshots with dense annotations across 55 UI element classes, including bounding boxes, semantic labels, text content, interactability flags, and reading order. Annotations were generated through automated DOM extraction, IoU-based filtering, and VLM-based refinement (Qwen3-VL-8B).

## Limitations

- Optimized for **web page screenshots**; performance on mobile or desktop application UIs may vary
- May struggle with very dense or highly dynamic UIs (e.g., complex dashboards with hundreds of elements)

## Citation

```bibtex
@misc{gurbuz2026movingsparsegroundingcomplete,
      title={ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision},
      author={A. Said Gurbuz and Sunghwan Hong and Ahmed Nassar and Marc Pollefeys and Peter Staar},
      year={2026},
      eprint={2602.14276},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2602.14276},
}
```