--- tags: - computer-vision - object-detection - multimodal - extended-reality - graphical-user-interface --- # Orienter Official code release for **“Look Before You Leap: Context-Sensitive GUI Grounding for Boosting Automated Extended Reality (XR) Testing.”** [Paper](https://doi.org/10.1145/3808134) · [Project page](https://sites.google.com/view/gui-orienter) · [Pipeline guide](PIPELINE_README.md) · [Environment](docs/ENVIRONMENT.md) · [Required assets](docs/ASSETS.md) · [Model manifest](docs/MODEL_MANIFEST.md) · [Release checklist](docs/RELEASE_CHECKLIST.md) Orienter detects context-sensitive, user-interactable GUI elements in XR screenshots. The public pipeline has three explicit stages: 1. an LMM combines application context and the current screenshot to mine interactable element descriptions; 2. APE-L\(_D\) grounds those descriptions, with the optional PII.5–PII.7 reflection loop described in the paper; 3. the evaluation tools convert, merge, filter, and score predictions. The repository contains source code, selected evaluation ground truth, and the APE-L\(_D\) checkpoint needed by the main detector. Dataset image payloads, private data, API credentials, experiment logs, generated predictions, and previous experimental results are deliberately excluded. ## Release status The offline control path and utilities are unit-tested. Maintainer validation has exercised one-image live paths for the default OpenRouter VLM, APE-L\(_D\), and the PII.5–PII.7 advisor loop without retaining generated artifacts in this public copy. Historical experiment outputs and validation result tables are not bundled. A fresh-clone paper-scale inference reproduction still requires the external assets in [docs/ASSETS.md](docs/ASSETS.md), especially the dataset image payload, provider access, and a validated CUDA/APE environment. The paper is published in PACMSE, Volume 3, Issue FSE, Article FSE127 ([DOI 10.1145/3808134](https://doi.org/10.1145/3808134)). Do not publish this directory until the repository license, third-party revision audit, and final public model/data links are complete. Machine-readable citation metadata is available in [`CITATION.cff`](CITATION.cff). ## Repository layout | Path | Purpose | | --- | --- | | `approach/run_vlm.py` | configurable screenshot-context and candidate generation | | `approach/run_ape.py` | configurable APE detection and optional reflection | | `approach/reflection.py` | paper-aligned PII.5, PII.6, and PII.7 loop | | `approach/config.py` | OpenRouter and paper model profiles | | `approach/method*.py` | legacy experiment entrypoints retained for provenance | | `approach/ovod/` | vendored APE, Grounding DINO, and related detector code | | `evaluation/tools/` | prediction conversion and validated shard merge | | `evaluation/` | detection, semantic, context, and interaction evaluation | | `dataset/` | dataset conversion and split-generation scripts | | `tests/` | offline regression tests for release-critical logic | ## Installation The supported public environment targets Python 3.10 or later so it can use a currently supported Pillow release with the latest security fixes: ```bash python -B -m venv .venv source .venv/bin/activate python -B -m pip install --upgrade pip python -B -m pip install -r requirements.txt ``` `requirements.txt` covers the release utilities and provider adapters. APE-L\(_D\) has a separate CUDA/PyTorch stack under `approach/ovod/APE`; the checkpoint is bundled at `approach/ovod/APE/ape_d_model_final.pth` and recorded in [docs/MODEL_MANIFEST.md](docs/MODEL_MANIFEST.md). The public tree intentionally excludes compiled APE extensions, build directories, and egg-info metadata, so build/install the APE extension inside the target CUDA environment before running Stage 2. See [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) and `environment.reference.yml` for the tested historical environment, a public compatibility starting point, build commands, and verification boundary. The reference YAML is not a cross-platform lockfile or a security attestation. The historical combined dependency snapshot remains at `approach/requirements.txt`, but it is not a clean lockfile. ## Model configuration The default provider is OpenRouter. In this project workspace, use `openrouter-run` so the key is injected by the local credential wrapper and never written to source files, logs, or shell history: ```bash openrouter-run python -B -m approach.run_vlm --help ``` For a public fresh clone outside this workspace, inject `OPENROUTER_API_KEY` through the runner's secret manager or process environment. Do not commit `.env`, shell history, logs, or generated manifests containing credential values. The default profile uses `openai/gpt-5.6-sol`, the current flagship model configured for this release. Override it without editing code: ```bash export OPENROUTER_MODEL=openai/gpt-5.6-sol ``` Paper-oriented profiles remain available through OpenRouter: | Profile | Model | | --- | --- | | `paper_openai` / `gpt4v` | `openai/gpt-4o-2024-08-06` | | `paper_claude` / `claude35sonnet` | `anthropic/claude-3.5-sonnet` | | `paper_gemini` / `gemini15pro` | `google/gemini-pro-1.5` | Profiles are defined in `approach/config.py`. Library callers can override the catalog with a YAML or JSON profile file; CLI users select a named profile and may override the default model through `OPENROUTER_MODEL`. The paper profiles preserve the evaluated identifiers for GPT-4o-2024-08-06, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Provider catalogs may retire historical routes; if one is unavailable, use a custom profile or direct vendor access and record the substitution instead of silently changing the paper profile. The client records only the credential variable name in run manifests; it never writes the credential value. ## Quick start The commands below show the complete public interface. They require the external files listed in [docs/ASSETS.md](docs/ASSETS.md). Keep screenshots, metadata, prompts, predictions, visualizations, and evaluation files outside the repository: ```bash export ORIENTER_DATA_DIR=/absolute/path/to/orienter_data export ORIENTER_RUN_DIR=/absolute/path/to/orienter_run mkdir -p "$ORIENTER_RUN_DIR" ``` Build a question manifest from mounted screenshots: ```bash python -B scripts/generate_questions.py \ --images-dir "$ORIENTER_DATA_DIR/images" \ --output "$ORIENTER_RUN_DIR/questions.jsonl" \ --prompt "Identify interactable elements." \ --metadata-template "$ORIENTER_RUN_DIR/app_metadata.json" ``` Screenshot filenames must match `_.`, for example `123_4.jpg`; additional underscores are not supported. The frame must be in `[0, 999]` and is zero-padded to three digits when deriving `image_id`. Fill every generated `app_name` and `app_description` field in `$ORIENTER_RUN_DIR/app_metadata.json` before Stage 1; blank or missing records fail closed. Validate the mounted inputs and the metadata coverage: ```bash python -B scripts/verify_assets.py \ --repo-root . \ --questions "$ORIENTER_RUN_DIR/questions.jsonl" \ --images-dir "$ORIENTER_DATA_DIR/images" \ --app-metadata-cache "$ORIENTER_RUN_DIR/app_metadata.json" ``` Generate candidate descriptions: ```bash openrouter-run python -B -m approach.run_vlm \ --questions "$ORIENTER_RUN_DIR/questions.jsonl" \ --images-dir "$ORIENTER_DATA_DIR/images" \ --output "$ORIENTER_RUN_DIR/candidates.jsonl" \ --profile default \ --app-metadata-cache "$ORIENTER_RUN_DIR/app_metadata.json" ``` `--app-metadata-cache` accepts JSON or JSONL records keyed by `app_id` and avoids live Steam page drift. When supplied, missing app IDs fail closed. Omit it only for exploratory runs where live Steam lookup is acceptable. ### Data privacy Stage 1 sends each source screenshot to the selected OpenRouter/provider model. When reflection is enabled, PII.5 additionally sends the bounding-box crop and PII.6 sends the annotated full scene. Do not run private, personally identifying, confidential, or otherwise unauthorized screenshots through a remote provider. Review the selected provider's retention and data-processing terms, obtain the necessary permission, and use an approved local/provider deployment when screenshots cannot leave your environment. Generated candidates, predictions, traces, and error files may also reveal image labels or local filenames; keep them outside the release repository. Ground candidates with APE-L\(_D\): ```bash python -B -m approach.run_ape \ --questions "$ORIENTER_RUN_DIR/questions.jsonl" \ --candidates "$ORIENTER_RUN_DIR/candidates.jsonl" \ --images-dir "$ORIENTER_DATA_DIR/images" \ --output "$ORIENTER_RUN_DIR/predictions.json" \ --visualization-dir "$ORIENTER_RUN_DIR/ape_visualizations" \ --ape-root approach/ovod/APE \ --ape-checkpoint ape_d_model_final.pth ``` The released checkpoint is SHA-256 verified before loading. A different checkpoint is rejected unless `--trust-custom-checkpoint` is supplied explicitly after independently verifying its source; PyTorch checkpoints are executable pickle-based artifacts and must not be loaded from an untrusted source. Enable the paper reflection loop explicitly because it incurs additional LMM calls: ```bash openrouter-run python -B -m approach.run_ape \ --questions "$ORIENTER_RUN_DIR/questions.jsonl" \ --candidates "$ORIENTER_RUN_DIR/candidates.jsonl" \ --images-dir "$ORIENTER_DATA_DIR/images" \ --output "$ORIENTER_RUN_DIR/predictions.json" \ --visualization-dir "$ORIENTER_RUN_DIR/ape_visualizations" \ --ape-checkpoint ape_d_model_final.pth \ --enable-reflection \ --reflection-profile default \ --max-reflection-iterations 10 ``` PII.5 sends the original scene and the bbox crop as separate images. The crop is neither enlarged nor flipped. PII.6 sends the full scene with current boxes, and PII.7 returns structured verification/refinement feedback. Rejected false positives are removed before the next detector call; verified detections are retained. ## Ranges, shards, resume, and merge Use either a range or deterministic modulo shards; the two selection modes are intentionally mutually exclusive. Output filenames receive a selection suffix automatically, so parallel jobs cannot overwrite each other. ```bash python -B -m approach.run_ape ... --start-index 0 --end-index 100 python -B -m approach.run_ape ... --num-shards 4 --shard-index 0 python -B -m approach.run_ape ... --num-shards 4 --shard-index 1 ``` Add `--resume` to continue from an existing output. Progress sidecars record completed images even when a valid inference produces zero detections. Merge shards only after validating overlap and coverage: ```bash python -B -m evaluation.tools.merge_predictions \ --inputs "$ORIENTER_RUN_DIR/predictions.shard00-of-04.json" \ "$ORIENTER_RUN_DIR/predictions.shard01-of-04.json" \ "$ORIENTER_RUN_DIR/predictions.shard02-of-04.json" \ "$ORIENTER_RUN_DIR/predictions.shard03-of-04.json" \ --questions "$ORIENTER_RUN_DIR/questions.jsonl" \ --output "$ORIENTER_RUN_DIR/predictions.merged.json" ``` The merge command fails on overlapping shard progress, duplicate detections, missing sidecars, or incomplete expected image coverage. ## Evaluation Convert prediction fields explicitly for each task: ```bash python -B -m evaluation.tools.to_pred \ --task semantics \ --input "$ORIENTER_RUN_DIR/predictions.merged.json" \ --output "$ORIENTER_RUN_DIR/predictions.semantics.json" ``` `--task` accepts `semantics`, `interactable`, `interaction`, or `all`; `all` writes separate outputs and never overwrites one task with another. The remaining metric scripts and expected ground-truth layouts are documented in [PIPELINE_README.md](PIPELINE_README.md). ## Tests Run the release-critical offline suite from the repository root: ```bash python -B scripts/smoke_control_path.py python scripts/run_tests.py ``` The deterministic control-path smoke uses a generated image plus injected provider/detector functions; it does not call a paid API or load APE weights. The unit tests are also offline. The final release gate additionally requires a real one-image inference, reflection smoke test, and evaluation smoke test after the external materials are supplied. ## Legacy experiment scripts `approach/method.py`, `method_fastuse.py`, `method_claude.py`, and `method_gemini.py` preserve historical branches and commented stage variants so experiment provenance is not lost. Their active APE paths now produce valid accumulated JSON and accept range/shard controls; `method.py` and `method_fastuse.py` also accept explicit data/checkpoint/output paths and reflection flags. For new runs, prefer `run_vlm.py` and `run_ape.py`. They make stage selection, paths, model profiles, app metadata caches, ranges, and shards configurable instead of requiring source edits. ## Data, licenses, and citation Dataset images, annotations, bundled weights, and vendored third-party code may use different licenses. Publish every external asset with its stable URL, version, SHA-256, license, and expected local path. See [docs/THIRD_PARTY.md](docs/THIRD_PARTY.md) and [docs/MODEL_MANIFEST.md](docs/MODEL_MANIFEST.md). The paper is published in *Proceedings of the ACM on Software Engineering*, Volume 3, Issue FSE, Article FSE127, pages 2858–2881, in 2026: ```bibtex @article{li2026look, author = {Shuqing Li and Binchang Li and Yepang Liu and Cuiyun Gao and Jianping Zhang and Shing-Chi Cheung and Michael R. Lyu}, title = {Look Before You Leap: Context-Sensitive GUI Grounding for Boosting Automated Extended Reality (XR) Testing}, journal = {Proceedings of the ACM on Software Engineering}, year = {2026}, volume = {3}, number = {FSE}, pages = {2858--2881}, articleno = {FSE127}, doi = {10.1145/3808134} } ``` `CITATION.cff` contains the same preferred paper citation. The top-level code license and final Hugging Face repository links remain publication decisions; licenses for the paper, data, model, and vendored source are separate and must not be inferred from one another.