Upload folder using huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,69 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
base_model: Qwen/Qwen3-VL-4B-Instruct
|
| 4 |
+
pipeline_tag: image-text-to-text
|
| 5 |
+
library_name: transformers
|
| 6 |
+
tags:
|
| 7 |
+
- gui-agent
|
| 8 |
+
- mobile-agent
|
| 9 |
+
- android-world
|
| 10 |
+
- vision-language-model
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
|
| 14 |
+
|
| 15 |
+
<p align="center">
|
| 16 |
+
<a href="https://arxiv.org/pdf/2603.24533"><img src="https://img.shields.io/badge/📄%20arXiv-UI--Voyager-red" alt="arXiv" /></a>
|
| 17 |
+
<a href="https://github.com/ui-voyager/UI-Voyager"><img src="https://img.shields.io/badge/🔗%20GitHub-UI--Voyager-green" alt="GitHub" /></a>
|
| 18 |
+
</p>
|
| 19 |
+
|
| 20 |
+
**UI-Voyager** is a novel two-stage self-evolving mobile GUI agent fine-tuned from [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct). Our 4B model achieves a **81.0%** success rate on the [AndroidWorld](https://github.com/google-research/android_world) benchmark, outperforming numerous recent baselines and **exceeding human-level performance**.
|
| 21 |
+
|
| 22 |
+
<p align="center">
|
| 23 |
+
<img src="https://github.com/ui-voyager/UI-Voyager/blob/main/assets/img/overview_uivoyager.png" alt="Overview" width="700">
|
| 24 |
+
<br>
|
| 25 |
+
<em>Overview of UI-Voyager performance on AndroidWorld</em>
|
| 26 |
+
</p>
|
| 27 |
+
|
| 28 |
+
## Highlights
|
| 29 |
+
|
| 30 |
+
- 🏆 **State-of-the-Art Performance**: Achieves **81.0%** success rate on AndroidWorld, surpassing human-level performance.
|
| 31 |
+
- 🔄 **Self-Evolving**: A two-stage training paradigm that learns from failed experiences to continuously improve.
|
| 32 |
+
- 📱 **Mobile GUI Agent**: Specialized for operating mobile device interfaces — recognizing UI elements, understanding functions, and completing tasks autonomously.
|
| 33 |
+
- 🧠 **Built on Qwen3-VL-4B**: Inherits the powerful vision-language capabilities of Qwen3-VL-4B-Instruct, including advanced visual perception, OCR, and multimodal reasoning.
|
| 34 |
+
|
| 35 |
+
## Model Details
|
| 36 |
+
|
| 37 |
+
| Attribute | Detail |
|
| 38 |
+
|-----------|--------|
|
| 39 |
+
| **Base Model** | [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) |
|
| 40 |
+
| **Parameters** | ~4B |
|
| 41 |
+
| **License** | MIT |
|
| 42 |
+
| **Task** | Mobile GUI Agent / Image-Text-to-Text |
|
| 43 |
+
| **Benchmark** | AndroidWorld |
|
| 44 |
+
| **Success Rate** | 81.0% |
|
| 45 |
+
|
| 46 |
+
## Evaluation on AndroidWorld
|
| 47 |
+
|
| 48 |
+
For full evaluation instructions using AndroidWorld with parallel emulators, please refer to our [GitHub repository](https://github.com/ui-voyager/UI-Voyager).
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
# Start parallel evaluation (4 emulators)
|
| 52 |
+
NUM_WORKERS=4 CONFIG_NAME=UI-Voyager ./run_android_world.sh
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Citation
|
| 56 |
+
|
| 57 |
+
If you find this work useful, please consider giving a star ⭐ and citation:
|
| 58 |
+
|
| 59 |
+
```bibtex
|
| 60 |
+
@misc{lin2026uivoyager,
|
| 61 |
+
title={UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience},
|
| 62 |
+
author={Zichuan Lin and Feiyu Liu and Yijun Yang and Jiafei Lyu and Yiming Gao and Yicheng Liu and Zhicong Lu and Yangbin Yu and Mingyu Yang and Junyou Li and Deheng Ye and Jie Jiang},
|
| 63 |
+
year={2026},
|
| 64 |
+
eprint={2603.24533},
|
| 65 |
+
archivePrefix={arXiv},
|
| 66 |
+
primaryClass={cs.LG},
|
| 67 |
+
url={https://arxiv.org/abs/2603.24533},
|
| 68 |
+
}
|
| 69 |
+
```
|