README.md

---
license: apache-2.0
base_model: meta-models/Muse-Glimmer-30B
library_name: peft
pipeline_tag: image-text-to-text
tags:
  - gui-grounding
  - computer-use
  - ui-agent
  - screenspot
  - gui-agent
  - lora
  - vision-language-model
  - click-prediction
---

# Muse-Glimmer-30B-GUI-Grounding-Fast

A lightweight **LoRA adapter** for Meta's [Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B) that turns it into a **fast, reliable GUI-grounding model**: given a screenshot and an instruction, it outputs the target element's bounding box **directly** as `[x1, y1, x2, y2]` — no verbose reasoning.

Built for **computer-use / GUI agents** that need an exact click coordinate *now*, at a fraction of the base model's inference cost.

## Why this exists

The base Muse-Glimmer-30B already *locates* UI elements well, but as an agentic model it "thinks out loud" — it buries the coordinate under ~200 tokens of reasoning and only emits a cleanly parseable box part of the time. That's expensive and unreliable inside an agent loop.

This adapter teaches it to answer with **only** the coordinate.

### Before vs. after (real example)

Instruction: **"close this window"** on a 960×540 screenshot (ground-truth box `[0.948, 0.144, 0.994, 0.207]`).

**Base Muse-Glimmer-30B** (rambles, coordinate buried in reasoning, often never emits a clean box):
```
 to=self ...Close this window. The window has minimize, maximize, close at top
right. Close is X. Coordinates. Approx top right... x ~ 0.96 to 0.99, y ~ 0.17
to 0.21. Let's give [958, 170, 985, 205] but need relative 0-1. So maybe
[0.958, 0.168, 0.985, 0.210]. Let's approximate. Better estimate: ...
```
(~200 tokens, and the parser only sometimes finds a usable box)

**This adapter** (one clean answer, ~5 tokens):
```
[0.958, 0.168, 0.985, 0.210]
```

Same location — but deterministic, instantly parseable, and ~10× cheaper per action.

## What's novel here

- **Reliability, not just capability.** Rather than chasing raw grounding accuracy, we target the *output-reliability* gap that makes strong VLMs hard to deploy as agents: we lift the usable-answer rate from **63% → 99.8%**.
- **Diversity beats steps.** Our first attempt (desktop-only, ~5k unique images) overfit and stalled at 70.5%. Swapping in **~40k unique screenshots** (desktop + mobile) — not more training steps — jumped accuracy to **87%** and fixed the weak domains. The lesson: for grounding LoRAs, *image diversity* is the dominant lever.
- **Tiny, surgical, cheap.** A single-node LoRA touching **0.7%** of params (vision tower frozen) converts a general agentic VLM into a specialized, production-ready GUI grounder — no full fine-tune, no RLHF.
- **First GUI-grounding adapter for Meta's Muse-Glimmer-30B** (to our knowledge), fully reproducible on open weights + open data.

## Results

### Headline — full ScreenSpot-v2 (all 1,272 samples)

| Model | Accuracy | Parse-rate |
|---|---|---|
| **Muse-Glimmer-30B-GUI-Grounding-Fast (this model)** | **88.1%** | **99.7%** |

Metric: predicted bounding-box center falls inside the ground-truth box (standard ScreenSpot click accuracy).

![Before vs after: accuracy and parse-rate](assets/01_before_after.png)

### The reliability win (diagnostic, 400-sample subset)

Under a plain "output the box" prompt, the **base** model reasons instead of emitting a clean coordinate:

| Setting | Accuracy | Parse-rate (emits a usable box) |
|---|---|---|
| Base Muse-Glimmer-30B (plain prompt) | 39.5% | 63% |
| + this adapter | 87.0% | 99.8% |

The 39.5% base figure is **harness-limited** — with its own agentic harness the base grounds at ~75%. Our adapter makes that ability come out **reliably** (parse 63% → ~100%) and directly.

### By platform & target type (full benchmark)

![Accuracy by platform and target type](assets/02_by_domain.png)

GitLab 93.2 · macOS 91.8 · iOS 90.3 · forum 88.6 · web/shop 88.2 · Android 87.2 · tool 86.7 · Windows 81.1  |  text 94.2 · icon 80.3

### How we got here — diversity beats steps

![Training progression](assets/03_training.png)

Run-1 (desktop-only, ~5k unique images) overfit and stalled at 70.5%. Run-2 (+mobile, **~40k unique images**) reached **88.1%** — the gain came from *image diversity*, not more training steps.

### Context vs other GUI-grounding models (ScreenSpot-v2)

![Comparison to other models](assets/04_comparison.png)

| Model | Params | ScreenSpot-v2 |
|---|---|---|
| SeeClick | ~9.6B | 54.0 |
| Qwen2-VL-7B | 7B | 66.9 |
| UGround-7B | 7B | 76.5 |
| OS-Atlas-7B | 7B | 87.1 |
| **This model (LoRA on Muse-Glimmer-30B)** | 30B | **88.1** |
| UGround-V1 | 7B | 89.4 |
| UI-TARS-7B | 7B | 91.6 |

We land **above OS-Atlas-7B and Qwen2-VL**, competitive with UGround-V1, just behind UI-TARS-7B — with a lightweight LoRA on a general-purpose model. *Other models' scores are published full-benchmark numbers; ours is our own eval harness on the full 1,272-sample set with the standard center-in-box metric — reproduce it with the code below. Not an officially verified leaderboard submission.*

## Usage

```python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForMultimodalLM
from peft import PeftModel

BASE = "meta-models/Muse-Glimmer-30B"
ADAPTER = "lemuralabs/Muse-Glimmer-30B-GUI-Grounding-Fast"

proc = AutoProcessor.from_pretrained(BASE)
model = AutoModelForMultimodalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

img = Image.open("screenshot.png").convert("RGB")
instruction = "the settings gear icon"
prompt = (f'In this UI screenshot, locate the element described as: "{instruction}". '
          'Respond with ONLY its bounding box as [x1,y1,x2,y2], each a 0-1 float '
          'relative to image width/height.')

msgs = [{"role": "user", "content": [{"type": "image", "image": img},
                                     {"type": "text", "text": prompt}]}]
inp = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                               return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inp, max_new_tokens=32, do_sample=False)
print(proc.tokenizer.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True))
# -> e.g. [0.912, 0.043, 0.958, 0.089]
```

Coordinates are normalized 0–1 (relative to image width/height). Take the box center as the click point.

## Training

- **Method:** LoRA (rank 32, α 64, dropout 0.05) on the language decoder only (`model.language_model.*`); vision encoder **frozen**.
- **Trainable params:** ~210M (0.7% of the 30B model).
- **Data:** ~160k GUI-grounding examples across ~40k unique screenshots, from the open [OS-Atlas](https://huggingface.co/datasets/OS-Copilot/OS-Atlas-data) corpus (desktop + mobile domains).
- **Schedule:** lr 5e-5, cosine, bf16, gradient checkpointing; best checkpoint at ~500 steps (early stopping — later steps overfit).

## Limitations

- Icons and dense/small targets remain the hardest (80% vs 94% for text).
- Trained/evaluated on English UI screenshots.
- Windows is the weakest platform (81%).
- Scores are from our own eval harness (full 1,272-sample ScreenSpot-v2, standard center-in-box metric), not an officially verified leaderboard submission.
- Requires `transformers >= 5.15` (base model architecture requirement).

## License & attribution

Released under **Apache 2.0**. This is a derivative adapter built on Meta's **Muse-Glimmer-30B** (Apache 2.0) and the open **OS-Atlas** dataset. Use is subject to the base model's [Usage Policy](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md). Credit to Meta for the base model.