🔍 YOLOv11l — UI Elements Detection

This model is a fine-tuned version of Ultralytics/YOLO11, trained to detect UI elements in macOS application screenshots.

It is part of the Screen2AX project — a research effort focused on generating accessibility metadata using computer vision.

⚠️ AGPL-3.0 licence — read before integrating. This model inherits the AGPL-3.0 licence from Ultralytics YOLO11. Embedding it in a product or service — including serving it behind a network API — requires open-sourcing the integrating application under AGPL-3.0, or obtaining a commercial Ultralytics Enterprise Licence. Details in License.

🧠 Task Overview

Task: Object Detection
Target: Individual UI elements

Supported Labels:

['AXButton', 'AXDisclosureTriangle', 'AXImage', 'AXLink', 'AXTextArea']

This model detects common interactive components typically surfaced in accessibility trees on macOS.

🗂 Dataset

Training data: macpaw-research/Screen2AX-Element

🚀 How to Use

🔧 Install Dependencies

pip install huggingface_hub ultralytics

🧪 Load the Model and Run Predictions

from huggingface_hub import hf_hub_download
from ultralytics import YOLO

# Download the model
model_path = hf_hub_download(
    repo_id="macpaw-research/yolov11l-ui-elements-detection",
    filename="ui-elements-detection.pt",
)

# Load and run prediction
model = YOLO(model_path)
results = model.predict("/path/to/your/image")

# Display result
results[0].show()

📜 License

This model is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0), as inherited from the original YOLOv11 base model and the Ultralytics training framework.

What this means in practice:

✅ Research, evaluation, and open-source (AGPL) projects — free to use, modify, and redistribute, provided derivative works are also released under AGPL-3.0.
⚠️ Commercial or closed-source use — AGPL-3.0 is a strong copyleft licence whose obligations also trigger on network use: shipping this model inside a proprietary application, or exposing it through a hosted API, requires releasing the integrating application's source code under AGPL-3.0.
💼 Need a proprietary integration? Ultralytics offers a commercial Enterprise Licence that lifts the AGPL obligations for YOLO models and their derivatives, including fine-tunes such as this one.

This summary is provided for convenience and is not legal advice.

🔗 Related Projects

✍️ Citation

If you use this model in your research, please cite the Screen2AX paper:

@misc{muryn2025screen2axvisionbasedapproachautomatic,
      title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation}, 
      author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
      year={2025},
      eprint={2507.16704},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.16704}, 
}