| # Phase Tracking |
|
|
| Date closed: `2026-03-25 UTC` |
|
|
| - Snapshot note: this Hugging Face snapshot does not contain `.git`, so a git commit hash is unavailable. |
| - Regression baselines: `/workspace/VLAarchtests/regression/baselines.md` |
| - Acceptance rule: only proxy metrics are used for phase acceptance. RLBench and PerAct2 remain integration-only checks. |
|
|
| ## Phase 0 |
|
|
| - Status: completed. |
| - Historical baseline artifacts are locked in `/workspace/VLAarchtests/regression/baselines.md`. |
| - Historical dummy benchmark reference: |
| - interaction `0.5278` |
| - backbone `0.5556` |
| - reveal `0.5417` |
| - Historical CLIP benchmark reference: |
| - interaction `0.3056` |
| - backbone `0.3333` |
| - reveal `0.2083` |
|
|
| ## Phase 1 |
|
|
| - Config: `proxy_interaction_r3d_stage1_dummy.yaml` |
| - Seeds: `13, 14, 15` |
| - Artifact roots: |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_dummy_seed13` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_dummy_seed14` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_dummy_seed15` |
| - Mean train time: `20.45 s` |
| - Mean peak GPU memory: `629.62 MB` |
| - 3-seed benchmark means: |
| - mean success: `0.5787` |
| - foliage success: `0.4444` |
| - bag success: `0.6111` |
| - cloth success: `0.6806` |
| - reocclusion rate: `0.0000` |
| - persistence horizon MAE: `1.9553` |
| - disturbance cost: `0.3649` |
| - planner top-1: `0.2832` |
| - planner regret: `0.0143` |
| - planner score/utility spearman: `0.2504` |
| - role collapse: `0.0000` |
| - proposal diversity: `0.0245` |
| - swap equivariance error: `0.00768` |
| - Ablations: |
| - `no_planner`: `0.5648` mean success, drop `0.0139` |
| - `no_role_symmetry`: `0.5833` mean success, delta `+0.0046` |
| - Gate decisions: |
| - hard success gate `>= 0.58`: fail by `0.0013` |
| - planner must matter: fail, `no_planner` drop is only `0.0139` |
| - planner top-1 `>= 0.30`: fail, measured `0.2832` |
| - role symmetry must matter: fail, `no_role_symmetry` is slightly better than full |
| - proposal collapse must not happen: pass, diversity stayed nonzero across all seeds |
| - Takeaway: the structure refactor improved over the historical interaction baseline (`0.5787` vs `0.5278`) and exceeded the historical dummy backbone baseline (`0.5556`), but it did not clear the phase-1 acceptance gates. |
|
|
| ## Phase 2 |
|
|
| - Config: `proxy_interaction_r3d_stage2_dummy.yaml` |
| - Seeds: `21, 22, 23` |
| - Artifact roots: |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_dummy_seed21` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_dummy_seed22` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_dummy_seed23` |
| - Mean train time: `20.76 s` |
| - Mean peak GPU memory: `639.39 MB` |
| - 3-seed benchmark means: |
| - mean success: `0.5463` |
| - foliage success: `0.4444` |
| - bag success: `0.5417` |
| - cloth success: `0.6528` |
| - reocclusion rate: `0.0121` |
| - persistence horizon MAE: `2.2358` |
| - disturbance cost: `0.3148` |
| - planner top-1: `0.3442` |
| - planner regret: `0.0208` |
| - planner score/utility spearman: `0.2397` |
| - belief calibration brier: `0.00842` |
| - reocclusion calibration brier: `0.2745` |
| - swap equivariance error: `0.00504` |
| - Ablations: |
| - `no_world_model`: `0.5463` mean success, drop `0.0000` |
| - `short_history`: `0.5463` mean success, delta `0.0000` |
| - Gate decisions: |
| - hard success gate `>= 0.60`: fail |
| - `no_world_model` must hurt: fail; the `2026-03-25` post-fix null-rollout rerun remained at `0.5463`, drop `0.0000` |
| - full memory must stop losing to short history: hard gate passes narrowly because full equals short-history; preferred gate fails because full does not beat short-history |
| - state metrics should improve over phase 1: fail, reocclusion rate increased (`0.0000 -> 0.0121`), persistence MAE worsened (`1.9553 -> 2.2358`), and calibration worsened |
| - Takeaway: the expanded state/memory path did not validate on the dummy proxy benchmark. Planner classification improved, but the post-fix null-rollout rerun still left mean success unchanged. |
|
|
| ## Phase 3 |
|
|
| - RGB-only compatibility configs: |
| - `proxy_interaction_r3d_stage1_clip.yaml` |
| - `proxy_interaction_r3d_stage2_clip.yaml` |
| - RGB-D config: `proxy_interaction_r3d_stage3_clip_rgbd.yaml` |
| - Artifact roots: |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_clip_seed7` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_clip_seed8` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage1_clip_seed9` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_clip_seed11` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_clip_seed12` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage2_clip_seed13` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed18` |
| - `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed19` |
| - RGB-only CLIP means: |
| - stage 1 clip mean success: `0.5324` |
| - stage 2 clip mean success: `0.4954` |
| - Stage 3 RGB-D means: |
| - mean train time: `145.93 s` |
| - mean peak GPU memory: `1952.12 MB` |
| - mean success: `0.5741` |
| - foliage success: `0.4861` |
| - bag success: `0.5417` |
| - cloth success: `0.6944` |
| - reocclusion rate: `0.0151` |
| - persistence horizon MAE: `1.7883` |
| - disturbance cost: `0.2258` |
| - planner top-1: `0.3265` |
| - planner regret: `0.0157` |
| - proposal diversity: `0.0270` |
| - swap equivariance error: `0.000094` |
| - `no_depth` ablation: |
| - mean success: `0.5231` |
| - absolute drop vs full: `0.0509` |
| - bag success drops `0.5417 -> 0.4722` |
| - foliage success drops `0.4861 -> 0.4167` |
| - Gate decisions: |
| - CLIP hard success gate `>= 0.37`: pass |
| - `no_depth` must hurt on at least one geometry-heavy proxy: pass |
| - no RGB-only regression: pass, both RGB-only CLIP configs still run and produce sane metrics |
| - Takeaway: the RGB-D path is the first phase that cleanly clears its acceptance gates. |
|
|
| ## Phase 4 |
|
|
| - Unit tests: |
| - command: `PYTHONPATH=/workspace/venv_r3d/lib/python3.11/site-packages:/workspace/VLAarchtests/code/reveal_vla_bimanual:/usr/local/lib/python3.11/dist-packages python -m pytest -q /workspace/VLAarchtests/tests` |
| - result: `10 passed` |
| - RLBench import/config smoke: |
| - artifact: `/workspace/VLAarchtests/artifacts/outputs/r3d/rlbench_smokes/smoke_test_output.txt` |
| - status: pass |
| - imports `rlbench`, `pyrep`, `yarr` all resolved |
| - camera contract preserved: `front`, `wrist_left`, `wrist_right` at `224x224` |
| - RLBench launch smoke: |
| - artifact stdout: `/workspace/VLAarchtests/artifacts/outputs/r3d/rlbench_smokes/launch_smoke_open_drawer.txt` |
| - artifact stderr: `/workspace/VLAarchtests/artifacts/outputs/r3d/rlbench_smokes/launch_smoke_open_drawer.stderr` |
| - status: pass |
| - `open_drawer` resolves to `RightOpenDrawer` |
| - finite `18`-D action, camera shapes `[224, 224, 3]`, no crash |
| - RLBench open-drawer rollout: |
| - artifact: `/workspace/VLAarchtests/artifacts/outputs/r3d/rlbench_open_drawer_r3d_rollout/rollout_eval.json` |
| - status: pass as integration |
| - no import errors |
| - no historical workspace path error string |
| - rollout JSON written |
| - mean success remains `0.0`, so this is plumbing evidence only |
| - PerAct2 13-task smoke: |
| - artifact summary JSON: `/workspace/VLAarchtests/artifacts/outputs/r3d/peract2_13_launch_smoke/launch_smoke_summary.json` |
| - artifact summary markdown: `/workspace/VLAarchtests/artifacts/outputs/r3d/peract2_13_launch_smoke/launch_smoke_summary.md` |
| - status: pass as integration |
| - all `13/13` tasks launched |
| - finite action check: `13/13` |
| - summary JSON written |
| - Integration caveat: |
| - full multi-task rollout in a single process is not reliable with this CoppeliaSim build. A direct batched `run_rlbench_rollout_eval` attempt hit a Qt/OpenGL segfault after repeated env recycle, and the subprocess-isolated full rollout sweep was too slow to be a reasonable smoke. The accepted PerAct2 artifact is therefore the launch/noop smoke, which matches the stated gate: launch stability, finite actions, and written summary. |
|
|
| ## Final Decision |
|
|
| - Phase 1: not accepted |
| - Phase 2: not accepted |
| - Phase 3: accepted |
| - Phase 4 integration: accepted |
|
|
| Overall status: the repo-preserving R3D-VLA refactor is implemented, verified, and benchmarked. The strongest positive result is the RGB-D CLIP phase. The structural planner/world-model claims are still not validated strongly enough on the dummy proxy benchmark to support a stronger paper claim without more work. |
|
|
