Instructions to use docling-project/ScreenVLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use docling-project/ScreenVLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="docling-project/ScreenVLM")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("docling-project/ScreenVLM")
model = AutoModelForImageTextToText.from_pretrained("docling-project/ScreenVLM")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use docling-project/ScreenVLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "docling-project/ScreenVLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "docling-project/ScreenVLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/docling-project/ScreenVLM

SGLang

How to use docling-project/ScreenVLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "docling-project/ScreenVLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "docling-project/ScreenVLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "docling-project/ScreenVLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "docling-project/ScreenVLM",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use docling-project/ScreenVLM with Docker Model Runner:
```
docker model run hf.co/docling-project/ScreenVLM
```

Saidgurbuz commited on Feb 11

Commit

5502c07

verified ·

1 Parent(s): 4f79cd7

Update README.md

Browse files

Files changed (1) hide show

README.md +1 -2

README.md CHANGED Viewed

@@ -219,7 +219,7 @@ print(f"\nTotal: {time.time() - start:.1f}s for {len(batched_inputs)} images")
 ## Training
-ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework on IBM's Blue Vela supercomputing cluster with NVIDIA H100 GPUs.
 **Training data**: [ScreenParse](https://huggingface.co/docling-project/screenparse) — 771K web page screenshots with dense annotations across 55 UI element classes, including bounding boxes, semantic labels, text content, interactability flags, and reading order. Annotations were generated through automated DOM extraction, IoU-based filtering, and VLM-based refinement (Qwen3-VL-8B).
@@ -228,7 +228,6 @@ ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM
 - Optimized for **web page screenshots**; performance on mobile or desktop application UIs may vary
 - Coordinate predictions are approximate — fine-grained pixel-level precision is not guaranteed
 - May struggle with very dense or highly dynamic UIs (e.g., complex dashboards with hundreds of elements)
-- Not designed for general image understanding — use [Granite Vision](https://huggingface.co/collections/ibm-granite/granite-vision-models-67b3bd4ff90c915ba4cd2800) for general-purpose vision tasks
 ## Citation

 ## Training
+ScreenVLM was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework with 16 NVIDIA H100 GPUs.
 **Training data**: [ScreenParse](https://huggingface.co/docling-project/screenparse) — 771K web page screenshots with dense annotations across 55 UI element classes, including bounding boxes, semantic labels, text content, interactability flags, and reading order. Annotations were generated through automated DOM extraction, IoU-based filtering, and VLM-based refinement (Qwen3-VL-8B).
 - Optimized for **web page screenshots**; performance on mobile or desktop application UIs may vary
 - Coordinate predictions are approximate — fine-grained pixel-level precision is not guaranteed
 - May struggle with very dense or highly dynamic UIs (e.g., complex dashboards with hundreds of elements)
 ## Citation