Image-Text-to-Text
Transformers
Safetensors
qwen2_5_vl
conversational
Eval Results
text-generation-inference
Instructions to use KDEGroup/UI-AGILE-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KDEGroup/UI-AGILE-3B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KDEGroup/UI-AGILE-3B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KDEGroup/UI-AGILE-3B") model = AutoModelForMultimodalLM.from_pretrained("KDEGroup/UI-AGILE-3B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KDEGroup/UI-AGILE-3B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KDEGroup/UI-AGILE-3B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KDEGroup/UI-AGILE-3B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KDEGroup/UI-AGILE-3B
- SGLang
How to use KDEGroup/UI-AGILE-3B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KDEGroup/UI-AGILE-3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KDEGroup/UI-AGILE-3B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KDEGroup/UI-AGILE-3B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KDEGroup/UI-AGILE-3B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KDEGroup/UI-AGILE-3B with Docker Model Runner:
docker model run hf.co/KDEGroup/UI-AGILE-3B
Add metadata tags and improve model card
#2
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,13 +1,39 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
| 4 |
# UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
|
| 5 |
|
| 6 |
<font size=4><div align='center' > [[📖 Paper](https://arxiv.org/abs/2507.22025)] [[🤗 Checkpoints](https://huggingface.co/KDEGroup/UI-AGILE)] [[🤗 Data](https://huggingface.co/datasets/KDEGroup/UI-AGILE-Data)] [[🤗 Daily Paper](https://huggingface.co/papers/2507.22025)] [[🚀 Github](https://github.com/KDEGroup/UI-AGILE)]</div></font>
|
| 7 |
|
| 8 |
## 🔥 Overview
|
| 9 |
|
| 10 |
-
UI-AGILE
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: transformers
|
| 4 |
+
pipeline_tag: image-text-to-text
|
| 5 |
---
|
| 6 |
+
|
| 7 |
# UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
|
| 8 |
|
| 9 |
<font size=4><div align='center' > [[📖 Paper](https://arxiv.org/abs/2507.22025)] [[🤗 Checkpoints](https://huggingface.co/KDEGroup/UI-AGILE)] [[🤗 Data](https://huggingface.co/datasets/KDEGroup/UI-AGILE-Data)] [[🤗 Daily Paper](https://huggingface.co/papers/2507.22025)] [[🚀 Github](https://github.com/KDEGroup/UI-AGILE)]</div></font>
|
| 10 |
|
| 11 |
## 🔥 Overview
|
| 12 |
|
| 13 |
+
UI-AGILE is a framework designed to enhance Graphical User Interface (GUI) agents at both training and inference stages. It addresses common challenges in Multimodal Large Language Models (MLLMs) such as reasoning designs, ineffective rewards, and visual noise.
|
| 14 |
+
|
| 15 |
+
### Key Features
|
| 16 |
+
- **Training Enhancements**:
|
| 17 |
+
- **Continuous Reward Function**: Incentivizes high-precision grounding.
|
| 18 |
+
- **"Simple Thinking" Reward**: Balances planning depth with execution speed and grounding accuracy.
|
| 19 |
+
- **Cropping-based Resampling**: Mitigates the sparse reward problem and improves learning on complex tasks.
|
| 20 |
+
- **Inference Enhancements**:
|
| 21 |
+
- **Decomposed Grounding with Selection**: Dramatically improves grounding accuracy on high-resolution displays by breaking the image into smaller, manageable parts.
|
| 22 |
+
|
| 23 |
+
UI-AGILE-7B achieves state-of-the-art grounding performance on benchmarks like ScreenSpot-Pro and ScreenSpot-v2 while maintaining strong general agent capabilities.
|
| 24 |
+
|
| 25 |
+
## ⭐️ Citation
|
| 26 |
+
|
| 27 |
+
If you find this project useful, please cite:
|
| 28 |
|
| 29 |
+
```bibtex
|
| 30 |
+
@misc{lian2025uiagileadvancingguiagents,
|
| 31 |
+
title={UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding},
|
| 32 |
+
author={Shuquan Lian and Yuhang Wu and Jia Ma and Zihan Song and Bingqi Chen and Xiawu Zheng and Hui Li},
|
| 33 |
+
year={2025},
|
| 34 |
+
eprint={2507.22025},
|
| 35 |
+
archivePrefix={arXiv},
|
| 36 |
+
primaryClass={cs.AI},
|
| 37 |
+
url={https://arxiv.org/abs/2507.22025},
|
| 38 |
+
}
|
| 39 |
+
```
|