| | ---
|
| | license: mit
|
| | library_name: ultralytics
|
| | tags:
|
| | - object-detection
|
| | - yolo
|
| | - gui
|
| | - ui-detection
|
| | - omniparser
|
| | pipeline_tag: object-detection
|
| | ---
|
| |
|
| | # GPA-GUI-Detector
|
| |
|
| | A YOLO-based GUI element detection model for detecting interactive UI elements (icons, buttons, etc.) on screen for GUI Process Automation. This model is finetuned from the [OmniParser](https://github.com/microsoft/OmniParser) ecosystem.
|
| |
|
| | ## Model
|
| |
|
| | The model weight file is `model.pt`. It is a YOLO model trained with the [Ultralytics](https://github.com/ultralytics/ultralytics) framework.
|
| |
|
| | ## Installation
|
| |
|
| | ```bash
|
| | pip install ultralytics
|
| | ```
|
| |
|
| | ## Usage
|
| |
|
| | ### Basic Inference
|
| |
|
| | ```python
|
| | from ultralytics import YOLO
|
| |
|
| | model = YOLO("model.pt")
|
| | results = model("screenshot.png")
|
| | ```
|
| |
|
| | ### Detection with Custom Parameters
|
| |
|
| | ```python
|
| | from ultralytics import YOLO
|
| | from PIL import Image
|
| |
|
| | # Load the model
|
| | model = YOLO("model.pt")
|
| |
|
| | # Run inference with custom confidence and image size
|
| | results = model.predict(
|
| | source="screenshot.png",
|
| | conf=0.05, # confidence threshold
|
| | imgsz=640, # input image size
|
| | iou=0.7, # NMS IoU threshold
|
| | )
|
| |
|
| | # Parse results
|
| | boxes = results[0].boxes.xyxy.cpu().numpy() # bounding boxes in [x1, y1, x2, y2]
|
| | scores = results[0].boxes.conf.cpu().numpy() # confidence scores
|
| |
|
| | # Draw results on image
|
| | img = Image.open("screenshot.png")
|
| | for box, score in zip(boxes, scores):
|
| | x1, y1, x2, y2 = box
|
| | print(f"Detected UI element at [{x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f}] (conf: {score:.2f})")
|
| |
|
| | # Or save the annotated image directly
|
| | results[0].save("result.png")
|
| | ```
|
| |
|
| | ### Integration with OmniParser
|
| |
|
| | ```python
|
| | import sys
|
| | sys.path.append("/path/to/OmniParser")
|
| |
|
| | from util.utils import get_yolo_model, predict_yolo
|
| | from PIL import Image
|
| |
|
| | model = get_yolo_model("model.pt")
|
| | image = Image.open("screenshot.png")
|
| |
|
| | boxes, confidences, phrases = predict_yolo(
|
| | model=model,
|
| | image=image,
|
| | box_threshold=0.05,
|
| | imgsz=640,
|
| | scale_img=False,
|
| | iou_threshold=0.7,
|
| | )
|
| |
|
| | for i, (box, conf) in enumerate(zip(boxes, confidences)):
|
| | print(f"Element {i}: box={box.tolist()}, confidence={conf:.2f}")
|
| | ```
|
| |
|
| | ## Example
|
| |
|
| | Detection results on a sample screenshot (1920x1080) from the [ScreenSpot-Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding) benchmark (`conf=0.05`, `iou=0.1`, `imgsz=1280`).
|
| |
|
| | **Input Screenshot**
|
| |
|
| | <p align="center">
|
| | <img src="images/example_input.png" width="80%" alt="Input Screenshot"/>
|
| | </p>
|
| |
|
| | <table>
|
| | <tr>
|
| | <th align="center">OmniParser V2</th>
|
| | <th align="center">GPA-GUI-Detector</th>
|
| | </tr>
|
| | <tr>
|
| | <td align="center"><img src="images/example_omniparser.png" width="92%" alt="OmniParser V2"/></td>
|
| | <td align="center"><img src="images/example_gpa.png" width="99%" alt="GPA-GUI-Detector"/></td>
|
| | </tr>
|
| | </table>
|
| |
|
| | ## License
|
| |
|
| | This model is released under the [MIT License](https://opensource.org/licenses/MIT).
|
| |
|
