README.md

---
license: mit
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- florence2
- GUI
- VLM
- GUI-Grounding
- visual-grounding
- custom_code
---

# PhoneUIAnchor-829M

PhoneUIAnchor-829M is a vision-language model for locating elements in mobile and
web interfaces. Given a screenshot and a natural-language description, it
returns the normalized center point of the target element. It can be used with
GUI agents, test automation, accessibility tools, and similar visual workflows.

This repository contains a complete, standalone checkpoint that can be loaded
directly with Transformers without additional model files.

## Capabilities

- **Intent grounding:** locate the control required to complete an action.
- **Description grounding:** locate an element from visible or semantic traits.
- **Function grounding:** locate a control from a detailed functionality
  description.
- **Cross-resolution output:** return normalized 0-999 coordinates that can be
  mapped to any screenshot size.
- **Local deployment:** run from a single 829M-parameter model without an
  external inference API.

## Technical Profile

| Property | Value |
| --- | --- |
| Architecture | Florence-2-large |
| Task | GUI element grounding |
| Parameters | 829M |
| Output contract | `<loc_x>,<loc_y>` |
| Weight format | Safetensors |
| Recommended precision | BF16 |
| Tested stack | Python 3.10, PyTorch 2.4.1, CUDA 12.1 |

## Installation

```bash
pip install -r requirements.txt
```

## Usage

The packaged Florence-2 implementation uses custom model code. Review the
included Python files and load with `trust_remote_code=True`.

```python
import re
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "lumimate/PhoneUIAnchor-829M"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).cuda().eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

image = Image.open("ui_screenshot.png").convert("RGB")
prompt = (
    'Where is the "Settings" element? '
    '(Output the center coordinates of the target)'
)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
    "cuda", dtype=torch.bfloat16
)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=16,
    )

text = processor.tokenizer.batch_decode(
    output_ids, skip_special_tokens=False
)[0]
match = re.search(r"<loc_(\d+)>,<loc_(\d+)>", text)
point = tuple(map(int, match.groups())) if match else None

x_px = point[0] / 999 * image.width
y_px = point[1] / 999 * image.height
print({"normalized": point, "pixels": (x_px, y_px)})
```

A command-line implementation is included in `inference_example.py`.

## Intended Use

PhoneUIAnchor-829M is intended for research and product prototyping in GUI
perception, agentic interaction, automated testing, and assistive interfaces.
Predictions should be validated before high-impact or irreversible automation.

## Limitations

- Development data emphasizes mobile user interfaces and may not cover every
  desktop, game, embedded, or highly customized UI style.
- Coordinate quality can degrade on very small, occluded, or visually
  ambiguous controls.
## Architecture and Licensing

PhoneUIAnchor-829M uses the Florence-2 architecture and is distributed under the
terms provided in `LICENSE`.