code/VLAarchtests4_root/PUBLIC_BENCHMARK_RESULTS.md

# Public Benchmark Results

All dates below refer to `2026-04-01 UTC`.

## Dense Occluded Retrieval Proxy

Benchmark:

- ManiSkill `PickClutterYCB-v1`

### Completed runs

- `reports/maniskill_pickclutter_smoke/public_benchmark_package_summary.json`
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.04`
- `reports/maniskill_pickclutter_smoke_v2/public_benchmark_package_summary.json`
  - `trunk = 0.04`
  - `noop = 0.32`
  - `active = 0.32`
  - not adapter-specific because `active == noop`
- `reports/maniskill_pickclutter_smoke_v3/public_benchmark_package_summary.json`
  - `trunk = 0.06`
  - `noop = 0.06`
  - `active = 0.06`
- `reports/maniskill_pickclutter_smoke_v4/public_benchmark_package_summary.json`
  - `trunk = 0.48`
  - `noop = 0.04`
  - `active = 0.04`
  - active intervened but regressed badly
- `reports/maniskill_pickclutter_smoke_v4_evalprobe_fromv3/public_benchmark_package_summary.json`
  - `trunk = 0.06`
  - `noop = 0.06`
  - `active = 0.62`
  - `delta = +0.56`
  - eval-probe only, not a clean retrain
- `reports/maniskill_pickclutter_smoke_v5/public_benchmark_package_summary.json`
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.04`
  - fairness-preserving retrain, but active still failed
- `reports/maniskill_pickclutter_smoke_v5_val_sweep/summary.json`
  - val-only planner sweep
  - `baseline_corrected = 0.00`
  - `soft_pref = 0.00`
  - `softer_pref = 0.625`
  - `retrieve_open = 0.625`
- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref/public_benchmark_package_summary.json`
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.62`
  - `delta = +0.58`
  - `95% CI = [0.44, 0.72]`
  - `intervention_rate = 1.0`
  - `non_base_selection_rate = 1.0`
- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref_gifs/`
  - full rerender of all `50` held-out seeds for `trunk_only_ft` and `adapter_active_ft`
  - includes `index.html`, `INDEX.md`, and `manifest.json`
  - rerender manifest reports `0` success mismatches against the saved benchmark json files

### Exact `smoke_v5` eval tuning carried to held-out

- `mode_preference_bonus = 0.75`
- `premature_retrieve_penalty = 0.5`
- `premature_insert_penalty = 0.25`
- `premature_maintain_penalty = 1.0`
- `occlusion_maintain_gap_min_access = 0.30`
- `occlusion_maintain_gap_min_visibility = 0.20`
- `retrieve_stage_access_threshold = 0.18`
- `retrieve_stage_reveal_threshold = 0.18`
- `retrieve_stage_support_threshold = 0.18`

## Bag Retrieval Proxy

Benchmark:

- ManiSkill public bridge basket retrieval proxy

### Completed runs

- `reports/maniskill_bag_bridge_smoke_v1/trunk_only_ft_seed17.json`
  - `0.32`
- `reports/maniskill_bag_bridge_smoke_v1/adapter_noop_seed17.json`
  - `0.00`
- `reports/maniskill_bag_bridge_smoke_v1/adapter_active_ft_seed17.json`
  - `0.48`

- `reports/maniskill_bag_bridge_smoke_v1/trunk_only_ft_seed23.json`
  - `0.48`
- `reports/maniskill_bag_bridge_smoke_v1/adapter_noop_seed23.json`
  - `0.08`
- `reports/maniskill_bag_bridge_smoke_v1/adapter_active_ft_seed23.json`
  - `0.00`

### Seed-23 validation sweep

- `reports/maniskill_bag_bridge_val_sweep_seed23/summary.json`

Configs:

- `default`
  - `trunk = 0.125`
  - `noop = 0.125`
  - `active = 0.00`
- `less_bonus`
  - `trunk = 0.125`
  - `noop = 0.125`
  - `active = 0.125`
  - intervention preserved
- `conservative`
  - `trunk = 0.125`
  - `noop = 0.125`
  - `active = 0.125`
  - intervention effectively disabled
- `low_bonus_high_thresh`
  - `trunk = 0.125`
  - `noop = 0.125`
  - `active = 0.125`
  - intervention effectively disabled

### Corrected held-out evals

- `reports/maniskill_bag_bridge_eval_less_bonus_seed17/public_benchmark_package_summary.json`
  - `trunk = 0.32`
  - `noop = 0.00`
  - `active = 0.48`
  - `delta = +0.16`
- `reports/maniskill_bag_bridge_eval_less_bonus_seed23/public_benchmark_package_summary.json`
  - `trunk = 0.48`
  - `noop = 0.08`
  - `active = 0.48`
  - `delta = +0.00`
- `reports/maniskill_bag_bridge_eval_less_bonus_2seed_manual_summary.json`
  - `trunk = 0.40`
  - `noop = 0.04`
  - `active = 0.48`
  - `delta = +0.08`
  - run-bootstrap CI `[0.00, 0.16]`

## Cloth Retrieval Proxy

Benchmark:

- ManiSkill public bridge cloth retrieval proxy

### Completed held-out seeds

- `seed17`
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.10`
  - `intervention = 0.3369`
  - `non_base = 0.2674`
- `seed23`
  - `trunk = 0.04`
  - `noop = 0.02`
  - `active = 0.02`
  - `intervention = 0.0`
  - `non_base = 0.0`
- `seed29`
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.04`
  - `intervention = 0.0`
  - `non_base = 0.0`

3-seed aggregate:

- `trunk = 0.0400`
- `noop = 0.0333`
- `active = 0.0533`
- `delta = +0.0133`

### Seed-23 cloth validation sweep

- `reports/maniskill_cloth_bridge_val_sweep_seed23/summary.json`

Configs:

- `default`
  - `trunk = 0.25`
  - `noop = 0.125`
  - `active = 0.125`
  - `intervention = 0.0`
- `low_thresh`
  - `active = 0.125`
  - `intervention = 0.2`
  - `non_base = 0.0667`
- `low_thresh_less_bonus`
  - `active = 0.125`
  - `intervention = 0.2`
  - `non_base = 0.0667`
- `very_low_thresh_less_bonus`
  - `active = 0.125`
  - `intervention = 1.0`
  - `non_base = 0.5333`

Interpretation:

- seed23 cloth was not recoverable by eval-side planner tuning alone

## Single-Seed Combined Proxy Suite

- `reports/public_proxy_suite_smoke_v1/combined_summary.json`

Single-seed summary:

- occlusion proxy: `+0.58`
- bag proxy: `+0.16`
- cloth proxy: `+0.06`
- macro delta: `+0.267`

This combined single-seed picture is useful historically, but the stronger current read is:

- occlusion: strong
- bag: modestly positive across corrected 2-seed evaluation
- cloth: weak/inconclusive across 3 seeds