PIPELINE_README.md

# Orienter pipeline guide

This guide maps the public commands to the method described in the paper. It distinguishes the configurable release path from historical experiment scripts and precomputed artifacts.

## Data contracts

The pipeline uses three small JSON formats.

Question manifest (`questions.jsonl`):

```json
{"question_id": 0, "image": "123_4.jpg", "image_id": 123004, "text": "Identify interactable elements."}
```

Candidate output (`candidates.jsonl`):

```json
{"question_id": 0, "text": {"objects": {"red button": "round red button with white text"}}, "model_id": "openai/gpt-5.6-sol"}
```

Detection output (`predictions.json`):

```json
[{"image_id": 123004, "category_id": "red button", "category_name": "red button", "bbox": [10, 20, 30, 40], "score": 0.91}]
```

Bounding boxes use COCO `x, y, width, height`. `image_id` is derived from `<app_id>_<frame>.ext` by zero-padding a frame in `[0, 999]` to three digits. Filenames must contain exactly one underscore between numeric app ID and numeric frame ID.

Keep all mounted inputs and generated artifacts outside the repository, then create the question manifest:

```bash
export ORIENTER_DATA_DIR=/absolute/path/to/orienter_data
export ORIENTER_RUN_DIR=/absolute/path/to/orienter_run
mkdir -p "$ORIENTER_RUN_DIR"

python -B scripts/generate_questions.py \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/questions.jsonl" \
  --prompt "Identify interactable elements." \
  --metadata-template "$ORIENTER_RUN_DIR/app_metadata.json"
```

Screenshot filenames must match `<numeric_app_id>_<numeric_frame>.<supported_ext>` and use a supported image extension. The manifest is sorted by derived `image_id` so repeated generation is deterministic. Fill the generated metadata template with a non-empty app name and description for every app ID before Stage 1.

## Stage 0: external assets

Before a live run, provide:

- XR screenshots and a matching question manifest;
- application metadata reachable from Steam, or a pinned JSON/JSONL metadata cache;
- the APE-L\(_D\) repository environment (the checkpoint itself is bundled);
- an OpenRouter credential for the selected LMM profile;
- the bundled ground-truth split matching the experiment, plus a reviewed semantic-embedding cache. Credentials are needed only when intentionally adding embeddings for previously unseen labels.

The repository includes the main checkpoint and selected evaluation ground truth. It intentionally excludes the screenshot payload, private context data, generated predictions, experiment results, and the semantic embedding cache. Exact requirements are listed in `docs/ASSETS.md`.

## Stage 1: context-sensitive candidate mining

Run:

```bash
openrouter-run python -B -m approach.run_vlm \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/candidates.jsonl" \
  --profile default \
  --app-metadata-cache "$ORIENTER_RUN_DIR/app_metadata.json"
```

`approach/run_vlm.py` selects the configured model profile, loads each screenshot, combines screenshot and application context with the research prompt, and atomically checkpoints JSONL responses. It accepts:

- `--start-index/--end-index` for a contiguous range;
- `--shard-index/--num-shards` for deterministic modulo sharding;
- `--resume` for completed `question_id` values;
- `--app-metadata-cache` for a pinned JSON/JSONL app metadata cache;
- `--ablation` for the historical no-interactability prompt.

Range and shard modes cannot be combined. Every selected output path receives a range/shard suffix.

The default profile uses OpenRouter. The optional paper profiles preserve the evaluated identifiers for GPT-4o-2024-08-06, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Because provider catalogs can retire historical routes, archive the resolved provider/model metadata with every reproduction; if a paper route is unavailable, supply a custom YAML/JSON profile or direct vendor adapter and document the substitution.

For reproducible runs, provide `--app-metadata-cache` or set `ORIENTER_APP_METADATA_CACHE`. The cache may be a JSON object keyed by app ID or JSONL records with `app_id`, `app_name`/`name`/`title`, and `app_description`/`description`. When this option is set, duplicate or missing app IDs fail before the provider call and the runner does not fall back to live Steam pages.

## Stage 2: APE-L_D visual grounding

Run:

```bash
python -B -m approach.run_ape \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --candidates "$ORIENTER_RUN_DIR/candidates.jsonl" \
  --images-dir "$ORIENTER_DATA_DIR/images" \
  --output "$ORIENTER_RUN_DIR/predictions.json" \
  --visualization-dir "$ORIENTER_RUN_DIR/ape_visualizations" \
  --ape-root approach/ovod/APE \
  --ape-config configs/LVISCOCOCOCOSTUFF_O365_OID_VGR_SA1B_REFCOCO_GQA_PhraseCut_Flickr30k/ape_deta/ape_deta_vitl_eva02_clip_vlf_lsj1024_cp_16x4_1080k.py \
  --ape-checkpoint ape_d_model_final.pth \
  --threshold 0.15
```

The runner:

1. accepts candidate `text` as a JSON object or JSON string;
2. converts object descriptions to APE referring expressions;
3. invokes APE once per selected screenshot;
4. enriches every result with `image_id` and `category_id`;
5. writes one atomic JSON document plus a progress sidecar;
6. records per-image failures separately without corrupting successful predictions.

This fixes the historical failure mode where `all_ape_res` was cleared inside the loop, multiple JSON documents were dumped into one file, and the raw `ape_item` was appended instead of the enriched item.

## PII.5–PII.7 reflection

Enable reflection with:

```bash
openrouter-run python -B -m approach.run_ape ... \
  --enable-reflection \
  --reflection-profile default \
  --max-reflection-iterations 10
```

The implementation follows the paper:

- **PII.5, mirroring-based regional verification:** crop each detected element inside its bbox and send the original scene plus crop as two separate images. The crop keeps its original pixel dimensions and orientation; it is not enlarged, resized, or flipped.
- **PII.6, unsuccessful-detection reflection:** send the full scene with current boxes so the advisor can identify missed elements and incorrect regions.
- **PII.7, advisor feedback:** require structured `verified`, `needs_refinement`, and `feedback` fields. False positives listed in `needs_refinement` are removed; verified detections are retained; feedback descriptions are grounded again by APE.

The loop stops when no refinement or feedback remains, or after at most 10 advisor rounds by default. Reflection is opt-in because it incurs additional API cost. If the advisor call fails in the configurable runner, the original APE detections are preserved and the reflection error is recorded.

## Parallel shards and validated merge

Example four-way detection:

```bash
for shard in 0 1 2 3; do
  python -B -m approach.run_ape ... --num-shards 4 --shard-index "$shard"
done
```

Each output is named `.shardNN-of-NN.json`; each progress sidecar also includes screenshots with zero detections. Merge with:

```bash
python -B -m evaluation.tools.merge_predictions \
  --inputs "$ORIENTER_RUN_DIR/predictions.shard00-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard01-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard02-of-04.json" \
           "$ORIENTER_RUN_DIR/predictions.shard03-of-04.json" \
  --questions "$ORIENTER_RUN_DIR/questions.jsonl" \
  --output "$ORIENTER_RUN_DIR/predictions.merged.json"
```

The merge fails closed on overlapping completed images, duplicate detections, missing progress files, and missing expected images. `--allow-incomplete` exists for deliberate partial diagnostics.

## Stage 3: task conversion and evaluation

Convert the merged list explicitly:

```bash
python -B -m evaluation.tools.to_pred \
  --task all \
  --input "$ORIENTER_RUN_DIR/predictions.merged.json" \
  --output "$ORIENTER_RUN_DIR/predictions.json"
```

This produces separate `semantics`, `interactable`, and `interaction` files. The task can no longer be changed accidentally by two consecutive assignments in source code.

The remaining evaluation sequence is:

1. `evaluation/filter_box.py` applies NMS and geometric filtering;
2. `evaluation/filter_test_prediction.py` selects the requested split;
3. `evaluation/evaluate_coco.py` computes modified COCO metrics;
4. `evaluation/context_eval.py` performs context-sensitive semantic matching;
5. `evaluation/autotest_study/` performs the RQ3 interaction simulation.

The historical shell scripts retain paper experiment names, but their data stems depend on external prediction/GT assets. Prefer explicit Python tool arguments for new runs.

### Offline semantic evaluation

Semantic evaluation can run without an API call by mounting the reviewed historical cache described in `evaluation/cache_manifest.json`:

```bash
export ORIENTER_EMBEDDING_CACHE=/absolute/path/to/embedding_dict.json
export ORIENTER_EMBEDDING_OFFLINE=1
export ORIENTER_EMBEDDING_READONLY=1
unset ZHIPU_API_KEY

python -B evaluation/evaluate_coco.py \
  -gt /absolute/path/to/semantics_test.json \
  -dt /absolute/path/to/predictions.json \
  -d s -s -l "$ORIENTER_RUN_DIR/semantic_eval.csv"
```

Offline mode fails before any provider call when a category is absent from the cache. Keep the reviewed cache read-only. If a genuinely new model introduces an uncached label, expand a separate working copy deliberately with `ZHIPU_API_KEY`, review it, record a new checksum, and then freeze that new version.

Verify the mounted historical cache against the frozen manifest before evaluation:

```bash
python -B scripts/verify_assets.py \
  --repo-root . \
  --skip-checkpoint \
  --embedding-cache /absolute/path/to/embedding_dict.json
```

The legacy `make_sample_gt.py` path keeps only image IDs present in a method's prediction file. It is retained for historical protocol compatibility, but evaluates different methods on different GT subsets. New comparisons must score every method against the same complete GT and report prediction coverage. Historical predictions, generated CSVs, and validation summaries are not bundled.

## Legacy experiment provenance

`approach/method*.py` contains commented branches for earlier datasets, ablations, machines, and model variants. Those comments were retained because a disabled block may represent another pipeline stage rather than dead code.

For release use:

- `run_vlm.py` replaces source-comment switching for screenshot-context generation;
- `run_ape.py` replaces source-comment switching for detector paths, ranges, shards, and reflection;
- explicit model profiles replace provider/model edits;
- the merge and conversion tools replace hand-edited evaluation state.

The legacy entrypoints remain available for provenance and now contain the APE accumulation fix and CLI range/shard controls, but they are not the recommended fresh-run interface.

## Verification boundary

Offline tests cover provider payloads, model aliases, PII.5 crop fidelity, false-positive removal, loop termination, APE accumulation/resume, shard isolation/merge, evaluation conversion, fail-closed semantic-cache access, and process-isolated evaluator temporary files.

The bundled tests verify evaluation mechanics without retaining prior prediction or metric artifacts. Paper-scale validation still requires fresh model inference, fixed-GT comparisons, and separately archived provenance. Selected evaluation ground truth is bundled, but the screenshot payload and authoritative public asset manifest are external.