| --- |
| license: apache-2.0 |
| datasets: |
| - docling-project/screenparse |
| tags: |
| - text-generation |
| - screen-parsing |
| - ui-understanding |
| - object-detection |
| - grounding |
| - web |
| - screentag |
| - docling |
| - granite |
| - mlx |
| - quantized |
| - apple-silicon |
| language: |
| - en |
| pipeline_tag: image-text-to-text |
| library_name: mlx |
| base_model: docling-project/ScreenVLM |
| --- |
| |
| # ScreenVLM MLX 4-bit |
|
|
| MLX 4-bit quantized version of [docling-project/ScreenVLM](https://huggingface.co/docling-project/ScreenVLM) for fast inference on Apple Silicon. |
|
|
| ## Model Details |
|
|
| - **Base model:** ScreenVLM (316M params, Idefics3 = SigLIP2-base-patch16-512 + Granite 165M) |
| - **Quantization:** 4-bit affine (7.654 bits/weight avg, vision encoder at higher precision) |
| - **Size:** 288 MB (vs 721 MB original float32) |
| - **License:** Apache 2.0 |
|
|
| ## Performance (Apple M4, 64GB) |
|
|
| | Metric | Value | |
| |--------|-------| |
| | Prompt processing | 747–1382 tok/s | |
| | Generation speed | 432–462 tok/s | |
| | Inference time | ~1.7s (172 tokens) | |
| | Peak memory | 1.1–1.2 GB | |
| | Model load | 1.1s | |
|
|
| ~500× faster than PyTorch CPU on the same hardware. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install mlx-vlm |
| ``` |
|
|
| ```python |
| from mlx_vlm import load, generate |
| from mlx_vlm.prompt_utils import apply_chat_template |
| from mlx_vlm.utils import load_config |
| |
| model, processor = load("olragon/ScreenVLM-MLX-4bit") |
| config = load_config("olragon/ScreenVLM-MLX-4bit") |
| |
| prompt = apply_chat_template(processor, config, "<screentag>", num_images=1) |
| output = generate(model, processor, prompt, image="screenshot.png", max_tokens=2048) |
| print(output) |
| ``` |
|
|
| ## Output Format (ScreenTag) |
|
|
| 55 UI element classes with normalized bounding boxes (0–500 grid): |
|
|
| ```xml |
| <button><loc_391><loc_46><loc_451><loc_49>Get started</button> |
| <tab><loc_582><loc_170><loc_633><loc_174>Tables</tab> |
| <logo><loc_62><loc_19><loc_182><loc_42>filament</logo> |
| <text><loc_73><loc_171><loc_427><loc_175>A cohesive set of building blocks</text> |
| ``` |
|
|
| Element types include: Button, Navigation Bar, Text Input, Link, Tab, Image, Video, Table, List, Card, Badge, Avatar, Alert, Search Bar, Logo, Heading, Code snippet, Checkbox, and more. |
|
|
| ## Conversion |
|
|
| ```bash |
| python -m mlx_vlm.convert \ |
| --model docling-project/ScreenVLM \ |
| --quantize --q-bits 4 \ |
| --mlx-path ./ScreenVLM-MLX-4bit |
| ``` |
|
|
| Requires `mlx-vlm >= 0.1.12`, `torch`, `torchvision` (for image processor conversion). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{gurbuz2026screenparse, |
| title={ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision}, |
| author={Gurbuz, A. Said and Hong, Sunghwan and Nassar, Ahmed and Pollefeys, Marc and Staar, Peter}, |
| booktitle={ICML}, |
| year={2026} |
| } |
| ``` |
|
|
| ## Acknowledgments |
|
|
| Original model by [IBM Research & ETH Zurich](https://huggingface.co/docling-project/ScreenVLM). MLX conversion by [olragon](https://huggingface.co/olragon). |
|
|
