---
license: apache-2.0
datasets:
- docling-project/screenparse
tags:
- text-generation
- screen-parsing
- ui-understanding
- object-detection
- grounding
- web
- screentag
- docling
- granite
- mlx
- quantized
- apple-silicon
language:
- en
pipeline_tag: image-text-to-text
library_name: mlx
base_model: docling-project/ScreenVLM
---

# ScreenVLM MLX 4-bit

MLX 4-bit quantized version of [docling-project/ScreenVLM](https://huggingface.co/docling-project/ScreenVLM) for fast inference on Apple Silicon.

## Model Details

- **Base model:** ScreenVLM (316M params, Idefics3 = SigLIP2-base-patch16-512 + Granite 165M)
- **Quantization:** 4-bit affine (7.654 bits/weight avg, vision encoder at higher precision)
- **Size:** 288 MB (vs 721 MB original float32)
- **License:** Apache 2.0

## Performance (Apple M4, 64GB)

| Metric | Value |
|--------|-------|
| Prompt processing | 747–1382 tok/s |
| Generation speed | 432–462 tok/s |
| Inference time | ~1.7s (172 tokens) |
| Peak memory | 1.1–1.2 GB |
| Model load | 1.1s |

~500× faster than PyTorch CPU on the same hardware.

## Usage

```bash
pip install mlx-vlm
```

```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("olragon/ScreenVLM-MLX-4bit")
config = load_config("olragon/ScreenVLM-MLX-4bit")

prompt = apply_chat_template(processor, config, "<screentag>", num_images=1)
output = generate(model, processor, prompt, image="screenshot.png", max_tokens=2048)
print(output)
```

## Output Format (ScreenTag)

55 UI element classes with normalized bounding boxes (0–500 grid):

```xml
<button><loc_391><loc_46><loc_451><loc_49>Get started</button>
<tab><loc_582><loc_170><loc_633><loc_174>Tables</tab>
<logo><loc_62><loc_19><loc_182><loc_42>filament</logo>
<text><loc_73><loc_171><loc_427><loc_175>A cohesive set of building blocks</text>
```

Element types include: Button, Navigation Bar, Text Input, Link, Tab, Image, Video, Table, List, Card, Badge, Avatar, Alert, Search Bar, Logo, Heading, Code snippet, Checkbox, and more.

## Conversion

```bash
python -m mlx_vlm.convert \
  --model docling-project/ScreenVLM \
  --quantize --q-bits 4 \
  --mlx-path ./ScreenVLM-MLX-4bit
```

Requires `mlx-vlm >= 0.1.12`, `torch`, `torchvision` (for image processor conversion).

## Citation

```bibtex
@inproceedings{gurbuz2026screenparse,
  title={ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision},
  author={Gurbuz, A. Said and Hong, Sunghwan and Nassar, Ahmed and Pollefeys, Marc and Staar, Peter},
  booktitle={ICML},
  year={2026}
}
```

## Acknowledgments

Original model by [IBM Research & ETH Zurich](https://huggingface.co/docling-project/ScreenVLM). MLX conversion by [olragon](https://huggingface.co/olragon).