initial release: 15-class MobileNetV3-small GUI element classifier

MANIFEST.md

@@ -0,0 +1,42 @@
+# Artifact Manifest — GUI Element Classifier
+
+Standalone bundle of a 15-class MobileNetV3-small GUI element type classifier, prepared for HuggingFace Hub release.
+
+## Files
+
+| File | sha256 | Purpose |
+|---|---|---|
+| `mobilenetv3_small.onnx` | `5e17d4c43de2927ca3f6ab56e46edf44d2d5ef991fa0afc5d1892bbbf65fbf05` | ONNX export, fixed batch=1, primary inference artifact |
+| `mobilenetv3_small.pth` | `a1176d43ce666e819d712892b0c8e47bdff99616f094c3e5e58d2ff886535292` | PyTorch state_dict source for the same weights — useful for fine-tuning further or re-exporting with dynamic batch axis |
+| `classes.json` | (small) | 15-class taxonomy + index ordering |
+| `inference_example.py` | (small) | Self-contained 100-line inference demo |
+| `README.md` | (small) | Model card |
+
+## Architecture & training
+
+- **Architecture:** MobileNetV3-small, 15-class output head.
+- **Training data:** A curated set of GUI element crops drawn primarily from Linux desktop UIs (~7000 crops across the 15 classes after consolidation).
+- **Training framework:** PyTorch with `torchvision.models.mobilenet_v3_small` backbone + cross-entropy loss on the 15-class head; ImageNet-pretrained backbone fine-tuned end-to-end.
+- **Held-out test set:** 1449 examples sampled from the training distribution (not strictly out-of-distribution).
+- **Export:** ONNX opset 17, fixed batch_size=1.
+- **Training-time metrics:** accuracy ≈ 0.72, macro-F1 ≈ 0.58, weighted-F1 ≈ 0.76. Per-class F1 details and the limits of these metrics are spelled out in `README.md`.
+
+## Framework versions used during this export
+
+- Python 3.11
+- ONNX Runtime 1.25.x (validated on the inference path)
+- PIL (Pillow) 10.4+
+- NumPy 1.26+
+
+## Build date
+
+- Weights frozen / exported: 2026-03 (training run). Bundle assembled for HF release: 2026-05-02.
+
+## Verification
+
+After download, verify the ONNX file hash matches:
+
+```bash
+sha256sum mobilenetv3_small.onnx
+# expected: 5e17d4c43de2927ca3f6ab56e46edf44d2d5ef991fa0afc5d1892bbbf65fbf05
+```

README.md

@@ -0,0 +1,138 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: onnx
+tags:
+- gui-element-classification
+- ui-classification
+- desktop-agents
+- agentic-ai
+- mobilenet-v3
+- computer-use
+- onnx
+pipeline_tag: image-classification
+---
+
+# GUI Element Classifier — MobileNetV3-small (15 classes)
+
+Lightweight (~6 MB) MobileNetV3-small ONNX classifier for **15 GUI element types**. CPU-friendly (~5 ms per crop, ONNX Runtime), designed as a deterministic preprocessing step before VLM-based GUI agent pipelines. No GPU required, no external dependencies beyond `onnxruntime`, `numpy`, and `pillow`.
+
+## What it's for
+
+You have a screenshot, a list of detected element bounding boxes (from any detector — YOLOv8, OWL-ViT, SAM-then-filter, accessibility tree, anything else), and you need cheap, deterministic per-element type labels (`button` vs `text_input` vs `slider` vs …) before passing the structured layout to a reasoning LLM. Drop this classifier into the pipeline as the typing layer:
+
+```
+[Screenshot]
+   ↓
+[Your detector] → list of bboxes
+   ↓
+[This classifier] → per-bbox type label + confidence
+   ↓
+[Your reasoning / action LLM] → reasons over typed elements, not pixels
+```
+
+The classifier is not a replacement for VLM captioning — it's the cheap deterministic layer that adds structure to your prompt so the LLM doesn't have to look at every region just to figure out what it is.
+
+## Classes (15)
+
+`button`, `checkbox`, `container`, `dropdown`, `icon_button`, `image`, `label`, `link`, `menu_item`, `scrollbar`, `slider`, `tab`, `text_input`, `toggle`, `unknown`
+
+The class indices in the model output (0..14) match the alphabetical ordering above. See [`classes.json`](classes.json) for the canonical list.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `mobilenetv3_small.onnx` | ONNX export (fixed batch=1). Primary inference artifact. |
+| `mobilenetv3_small.pth` | PyTorch state_dict for those who want to fine-tune further or re-export with dynamic axes. |
+| `classes.json` | Class names + ordering. |
+| `inference_example.py` | 100-line self-contained demo. `pip install onnxruntime numpy pillow` then `python inference_example.py crop.png`. |
+
+## Quick start
+
+```python
+from PIL import Image
+import numpy as np
+import onnxruntime as ort
+
+CLASSES = ['button','checkbox','container','dropdown','icon_button',
+           'image','label','link','menu_item','scrollbar',
+           'slider','tab','text_input','toggle','unknown']
+
+MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
+STD  = np.array([0.229, 0.224, 0.225], dtype=np.float32)
+
+def preprocess(img: Image.Image) -> np.ndarray:
+    img = img.convert("RGB")
+    w, h = img.size
+    m = max(w, h)
+    pad = Image.new("RGB", (m, m), (128, 128, 128))
+    pad.paste(img, ((m - w) // 2, (m - h) // 2))
+    arr = np.array(pad.resize((224, 224), Image.BILINEAR), dtype=np.float32) / 255.0
+    arr = (arr - MEAN) / STD
+    return arr.transpose(2, 0, 1)[None, :, :, :].astype(np.float32)
+
+sess = ort.InferenceSession("mobilenetv3_small.onnx",
+                            providers=["CPUExecutionProvider"])
+crop = Image.open("button.png")
+logits = sess.run(None, {sess.get_inputs()[0].name: preprocess(crop)})[0]
+probs = np.exp(logits - logits.max()) / np.exp(logits - logits.max()).sum()
+idx = int(probs.argmax())
+print(CLASSES[idx], float(probs[0, idx]))
+```
+
+## Preprocessing (must-match-or-quality-degrades)
+
+1. **PadToSquare with gray (128, 128, 128)** on the shorter axis.
+2. **Resize to 224x224** with `Image.BILINEAR`.
+3. `array / 255.0` → float32 in [0, 1].
+4. **ImageNet normalize**: `mean=[0.485, 0.456, 0.406]`, `std=[0.229, 0.224, 0.225]`.
+5. Transpose HWC → CHW. Add batch dim.
+
+The model is sensitive to all five steps — wrong pad colour, BICUBIC instead of BILINEAR, or skipping the ImageNet stats will degrade accuracy noticeably.
+
+## Performance
+
+Aggregate metrics from the training-time held-out evaluation:
+
+| Metric | Value |
+|---|---|
+| Test set | 1449 examples (sampled from the training distribution, **not strictly out-of-distribution** — see Limitations) |
+| Accuracy | ~0.72 |
+| Macro F1 | ~0.58 |
+| Weighted F1 | ~0.76 |
+
+Weighted F1 is dominated by the `icon_button` class (~50% of test support, F1=0.78). Per-class F1s vary widely — `tab` and `text_input` clear 0.95+, `container` and `slider` lag at 0.15-0.25. **If your domain skews heavily to specific classes, expect class-imbalance effects.**
+
+**Latency:** ~5 ms per crop on a modern x86 laptop CPU (Intel i7-12th gen, single thread). The shipped ONNX is fixed batch_size=1 for the simplest possible drop-in; if you're processing >100 crops per screenshot, re-export with dynamic axes from the included `.pth` for batching.
+
+## Limitations & honest scope
+
+- **Training-time test-set metrics are not a tight estimate of accuracy on your domain.** The aggregate numbers above (acc ~0.72, weighted F1 ~0.76) come from a single held-out split **sampled from the training distribution — this is not strictly out-of-distribution evaluation**. On a domain that differs from the training mix (web vs desktop, different OS look-and-feel, dark mode vs light mode, custom design systems), expect a meaningful gap from these numbers. Validate on your own labelled crops before depending on the figures.
+- **Per-class F1 varies widely.** `tab` and `text_input` clear 0.95+, `icon_button` sits around 0.78 (and dominates weighted F1 because of class imbalance — ~50% of test support), while `container` and `slider` lag at 0.15-0.25. The aggregate numbers (acc / macro / weighted F1) hide this variance — read the per-class story above before trusting the headline.
+- **Linux desktop bias.** Training data skews toward Linux desktop UIs (XFCE / GTK toolkits / Firefox / Mousepad / Thunar / terminal). Web pages, macOS, Windows 11, and mobile UIs are likely under-represented and may need domain adaptation.
+- **`unknown` is a real class.** When the classifier produces `unknown` with high confidence, the input is genuinely ambiguous (small icon with no clear visual identity); don't paper over it with `argmax-but-skip-unknown` logic.
+- **`container` and `slider` underperform** at training-time evaluation. Consider using bbox geometry as a sanity check (containers are large, sliders are wide-and-thin) alongside the model rather than trusting it alone for those two classes.
+- **Single-label argmax.** No multi-class output. If a region could legitimately be both `tab` and `button` (some apps style tabs as buttons), the model picks one.
+- **Fixed batch_size=1 in the shipped ONNX.** For high-throughput scenarios, re-export from the included `.pth` with dynamic axes (`torch.onnx.export(..., dynamic_axes={'input': {0: 'batch'}})`).
+- **ImageNet preprocessing is assumed.** The model was trained against the standard ImageNet mean/std + PadToSquare-with-gray. Substituting different normalization will silently degrade results.
+
+## License
+
+Apache-2.0. The MobileNetV3-small architecture itself was originally introduced by Google Research; this export uses the open architecture and re-trained weights.
+
+## Citation
+
+```
+@misc{gui_element_classifier_mobilenetv3_2026,
+  author = {Diogo Neno},
+  title  = {15-class MobileNetV3-small GUI Element Classifier},
+  year   = {2026},
+  url    = {https://huggingface.co/diogoneno/gui-element-classifier},
+}
+```
+
+## Changelog
+
+- `2026-05-02` — Initial public release.

classes.json

@@ -0,0 +1,21 @@
+{
+  "classes": [
+    "button",
+    "checkbox",
+    "container",
+    "dropdown",
+    "icon_button",
+    "image",
+    "label",
+    "link",
+    "menu_item",
+    "scrollbar",
+    "slider",
+    "tab",
+    "text_input",
+    "toggle",
+    "unknown"
+  ],
+  "n_classes": 15,
+  "ordering": "alphabetical (model output index 0..14 corresponds to this list in order)"
+}

inference_example.py

@@ -0,0 +1,105 @@
+"""Self-contained inference example for the 15-class UI-element classifier.
+
+Run:
+    pip install onnxruntime numpy pillow
+    python inference_example.py path/to/element_crop.png
+
+Designed to drop into an LLM-orchestration loop where you have a screenshot,
+a list of detected element bounding boxes (from any detector — YOLOv8, OWL-ViT,
+SAM-then-filter, accessibility tree, etc.), and you need cheap, deterministic
+per-element type labels before passing them to a reasoning LLM.
+
+Inference is CPU-friendly (~5 ms per crop on a modern x86 laptop). Use it as a
+'helper' that adds structure to the orchestrator's prompt — e.g., 'click the
+text_input near label "Username"' — instead of paying VLM tokens to look at
+every crop.
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+from PIL import Image
+
+HERE = Path(__file__).parent
+ONNX_PATH = HERE / "mobilenetv3_small.onnx"
+CLASSES = json.loads((HERE / "classes.json").read_text())["classes"]
+
+IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
+IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
+
+
+def pad_to_square(img: Image.Image) -> Image.Image:
+    """Pad shorter side with gray (128, 128, 128) — must match training transform."""
+    w, h = img.size
+    m = max(w, h)
+    out = Image.new("RGB", (m, m), (128, 128, 128))
+    out.paste(img, ((m - w) // 2, (m - h) // 2))
+    return out
+
+
+def preprocess(img: Image.Image) -> np.ndarray:
+    """PadToSquare -> Resize 224x224 BILINEAR -> /255 -> ImageNet normalize -> CHW."""
+    img = pad_to_square(img.convert("RGB"))
+    img = img.resize((224, 224), Image.BILINEAR)
+    arr = np.array(img, dtype=np.float32) / 255.0
+    arr = (arr - IMAGENET_MEAN) / IMAGENET_STD
+    arr = arr.transpose(2, 0, 1)
+    return arr[None, :, :, :].astype(np.float32)  # (1, 3, 224, 224)
+
+
+def softmax(x: np.ndarray) -> np.ndarray:
+    e = np.exp(x - np.max(x, axis=1, keepdims=True))
+    return e / np.sum(e, axis=1, keepdims=True)
+
+
+def classify(crop_path: str | Path) -> dict:
+    """Classify a single element crop. Returns label, confidence, full score map."""
+    so = ort.SessionOptions()
+    so.intra_op_num_threads = 4
+    sess = ort.InferenceSession(str(ONNX_PATH), sess_options=so, providers=["CPUExecutionProvider"])
+    img = Image.open(crop_path)
+    batch = preprocess(img)
+    logits = sess.run(None, {sess.get_inputs()[0].name: batch})[0]
+    probs = softmax(logits)[0]
+    idx = int(np.argmax(probs))
+    return {
+        "label": CLASSES[idx],
+        "confidence": float(probs[idx]),
+        "scores": {c: float(probs[i]) for i, c in enumerate(CLASSES)},
+    }
+
+
+def classify_batch(crop_paths: list[str | Path]) -> list[dict]:
+    """Convenience: per-crop loop. The shipped ONNX is fixed batch_size=1.
+
+    For higher throughput on large batches, re-export with dynamic axes and
+    run a single batched session.run() — kept simple here for clarity.
+    """
+    so = ort.SessionOptions()
+    so.intra_op_num_threads = 4
+    sess = ort.InferenceSession(str(ONNX_PATH), sess_options=so, providers=["CPUExecutionProvider"])
+    results = []
+    for p in crop_paths:
+        img = Image.open(p)
+        batch = preprocess(img)
+        logits = sess.run(None, {sess.get_inputs()[0].name: batch})[0]
+        probs = softmax(logits)[0]
+        idx = int(np.argmax(probs))
+        results.append({
+            "label": CLASSES[idx],
+            "confidence": float(probs[idx]),
+        })
+    return results
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("usage: python inference_example.py <crop.png> [<crop2.png> ...]")
+        sys.exit(1)
+    for path in sys.argv[1:]:
+        result = classify(path)
+        print(f"{path}: {result['label']} (confidence={result['confidence']:.3f})")

mobilenetv3_small.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e17d4c43de2927ca3f6ab56e46edf44d2d5ef991fa0afc5d1892bbbf65fbf05
+size 6142536

mobilenetv3_small.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1176d43ce666e819d712892b0c8e47bdff99616f094c3e5e58d2ff886535292
+size 6266042