2026-03-25 runpod handoff update

Upload updated code, tests, environment recreation files, generated proxy datasets, new checkpoints, and raw result artifacts from the 2026-03-25 /workspace runpod session.

README.md

@@ -9,195 +9,99 @@ tags:
 
 # VLAarchtests
 
-Bundle uploaded from the `/workspace` runpod session dated `2026-03-25 UTC`.
+Update uploaded from the `/workspace` runpod session dated `2026-03-25 UTC`.
 
-## Top-Level Contents
+## Updated Paths
 
 - `code/reveal_vla_bimanual/`
-  - current project code
-- `artifacts/outputs/r3d/`
-  - R3D-VLA proxy checkpoints, benchmarks, diagnostics, RLBench outputs, and PerAct2 smoke artifacts
-- `regression/baselines.md`
-  - locked historical baselines from the downloaded snapshot
-- `results/phase_tracking.md`
-  - phase-by-phase gate accounting and acceptance status
 - `tests/`
-  - unit tests for RGB, RGB-D, planner, memory, world-model, and dataset contracts
 - `environment/`
-  - same-machine setup helpers, env exports, runtime env vars, hardware snapshot, and upstream revision notes
-- `MODEL_INDEX.md`
-  - checkpoint/result index for the current R3D bundle
+- `artifacts/data/reveal_proxy/`
+- `artifacts/outputs/r3d_handoff/`
+- `artifacts/outputs/r3d_handoff_phase/`
+- `results/2026-03-25-runpod/`
 
-## Work Completed In This Bundle
+## Primary Source Changes
 
-### Architecture and Training Changes
-
-- Added the repo-preserving R3D-VLA refactor with updates in:
+- Geometry path and camera-pose propagation updates:
   - `code/reveal_vla_bimanual/models/backbones.py`
   - `code/reveal_vla_bimanual/models/multiview_fusion.py`
+  - `code/reveal_vla_bimanual/models/policy.py`
+- Spatial memory and world-model updates:
   - `code/reveal_vla_bimanual/models/observation_memory.py`
   - `code/reveal_vla_bimanual/models/reveal_head.py`
   - `code/reveal_vla_bimanual/models/world_model.py`
+- Semantic candidate and planner updates:
   - `code/reveal_vla_bimanual/models/action_decoder.py`
   - `code/reveal_vla_bimanual/models/planner.py`
-  - `code/reveal_vla_bimanual/models/policy.py`
+- Loss, dataset, and simulator updates:
   - `code/reveal_vla_bimanual/train/losses.py`
-  - `code/reveal_vla_bimanual/train/smoke_checks.py`
   - `code/reveal_vla_bimanual/sim_reveal/dataset.py`
   - `code/reveal_vla_bimanual/sim_reveal/procedural_envs.py`
-- Added new configs:
-  - `proxy_interaction_r3d_stage1_dummy.yaml`
-  - `proxy_interaction_r3d_stage1_clip.yaml`
-  - `proxy_interaction_r3d_stage2_dummy.yaml`
-  - `proxy_interaction_r3d_stage2_clip.yaml`
-  - `proxy_interaction_r3d_stage3_clip_rgbd.yaml`
-  - `proxy_interaction_r3d_ablation_nodepth.yaml`
-  - `proxy_interaction_r3d_ablation_noplanner.yaml`
-  - `proxy_interaction_r3d_ablation_nowm.yaml`
-  - `proxy_interaction_r3d_ablation_norolesym.yaml`
-
-### Evaluation and Integration Changes
-
-- Completed the requested proxy matrix:
-  - stage 1 dummy, 3 seeds
-  - stage 1 dummy `no_planner`
-  - stage 1 dummy `no_role_symmetry`
-  - stage 2 dummy, 3 seeds
-  - stage 2 dummy `no_world_model`
-  - stage 2 dummy `short_history`
-  - stage 1 clip, 3 seeds
-  - stage 2 clip, 3 seeds
-  - stage 3 clip RGB-D, 3 seeds
-  - stage 3 clip RGB-D `no_depth`
-- Completed RLBench integration artifacts:
-  - import/config smoke
-  - `open_drawer` launch smoke
-  - `open_drawer` rollout with JSON output
-- Added a dedicated PerAct2 13-task launch smoke harness:
-  - `code/reveal_vla_bimanual/eval/run_peract2_launch_smoke.py`
-  - `code/reveal_vla_bimanual/sim_rlbench/launch_smoke.py` now records finite-action checks
-
-### Tests
-
-- Full local test suite result:
-  - `10 passed`
-
-## Same-Machine Setup Files
-
-- `environment/setup_same_machine.sh`
-- `environment/validate_same_machine.sh`
-- `environment/run_peract2_13_rollouts.sh`
-- `environment/runtime_env_vars.sh`
-- `environment/hardware_snapshot.txt`
-- `environment/glxinfo_B.txt`
-- `environment/upstream_revisions.txt`
-- `environment/system_packages_same_machine.txt`
-- `environment/rlbench_env_export.yaml`
-- `environment/rlbench_env_explicit.txt`
-- `environment/rlbench_pip_freeze.txt`
-- `environment/reveal_env_export.yaml`
-- `environment/reveal_env_explicit.txt`
-- `environment/reveal_pip_freeze.txt`
-
-## Raw Proxy Benchmark Matrix
-
-| Run | Seeds | Mean success | foliage | bag | cloth | Reocclusion | Persistence MAE | Disturbance | Planner top-1 | Proposal diversity | Swap error |
-| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| stage1 dummy full | `13,14,15` | 0.5787 | 0.4444 | 0.6111 | 0.6806 | 0.0000 | 1.9553 | 0.3649 | 0.2832 | 0.0245 | 0.007680 |
-| stage1 dummy `no_planner` | `13,14,15` | 0.5648 | 0.4306 | 0.5972 | 0.6667 | 0.0000 | 1.9599 | 0.3765 | n/a | n/a | n/a |
-| stage1 dummy `no_role_symmetry` | `13,14,15` | 0.5833 | 0.4583 | 0.6111 | 0.6806 | 0.0000 | 1.9475 | 0.3547 | n/a | n/a | n/a |
-| stage2 dummy full | `21,22,23` | 0.5463 | 0.4444 | 0.5417 | 0.6528 | 0.0121 | 2.2358 | 0.3148 | 0.3442 | 0.0245 | 0.005036 |
-| stage2 dummy `no_world_model` | `21,22,23` | 0.5463 | 0.4444 | 0.5417 | 0.6528 | 0.0027 | 2.3600 | 0.3287 | n/a | n/a | n/a |
-| stage2 dummy `short_history` | `21,22,23` | 0.5463 | 0.4444 | 0.5417 | 0.6528 | 0.0121 | 2.2349 | 0.3148 | n/a | n/a | n/a |
-| stage1 clip full | `7,8,9` | 0.5324 | 0.4306 | 0.5278 | 0.6389 | 0.0244 | 1.3636 | 0.2808 | 0.2676 | 0.0217 | 0.000155 |
-| stage2 clip full | `11,12,13` | 0.4954 | 0.3889 | 0.4583 | 0.6389 | 0.0117 | 2.3198 | 0.2722 | 0.2693 | 0.0216 | 0.000186 |
-| stage3 clip RGB-D full | `17,18,19` | 0.5741 | 0.4861 | 0.5417 | 0.6944 | 0.0151 | 1.7883 | 0.2258 | 0.3265 | 0.0270 | 0.000094 |
-| stage3 clip RGB-D `no_depth` | `17,18,19` | 0.5231 | 0.4167 | 0.4722 | 0.6806 | 0.0198 | 2.0491 | 0.2548 | n/a | n/a | n/a |
-
-Full artifact roots are indexed in `MODEL_INDEX.md`.
+- Evaluation and RLBench tooling updates:
+  - `code/reveal_vla_bimanual/eval/run_reveal_benchmark.py`
+  - `code/reveal_vla_bimanual/eval/run_teacher_audit.py`
+  - `code/reveal_vla_bimanual/eval/run_rlbench_rollout_eval.py`
+  - `code/reveal_vla_bimanual/eval/run_peract2_task_sweep.py`
+  - `code/reveal_vla_bimanual/eval/compare_rlbench_sweeps.py`
+  - `code/reveal_vla_bimanual/scripts/run_rlbench_handoff_eval.sh`
 
-Note: the `stage2 dummy no_world_model` row above reflects the `2026-03-25` post-fix null-rollout rerun. Pre-fix copies are retained as `reveal_benchmark_pre_null_rollout_fix.json` and `reveal_benchmark_pre_null_rollout_fix.md` under each `benchmark_no_world_model` seed directory.
+## Validation
 
-## Raw Training Summaries
+- Test command:
+  - `PYTHONPATH=/workspace/VLAarchtests_work/code/reveal_vla_bimanual python -m pytest -q /workspace/VLAarchtests_work/tests`
+- Result:
+  - `33 passed`
 
-| Run | Mean train time (s) | Mean peak GPU memory (MB) |
-| --- | ---: | ---: |
-| stage1 dummy full | 20.45 | 629.62 |
-| stage2 dummy full | 20.76 | 639.39 |
-| stage1 clip full | 156.16 | 1908.92 |
-| stage2 clip full | 141.55 | 1902.54 |
-| stage3 clip RGB-D full | 145.93 | 1952.12 |
-
-## Raw RLBench Outputs
-
-### Import And Launch Smokes
-
-- import/config smoke file:
-  - `artifacts/outputs/r3d/rlbench_smokes/smoke_test_output.txt`
-- `open_drawer` launch smoke files:
-  - `artifacts/outputs/r3d/rlbench_smokes/launch_smoke_open_drawer.txt`
-  - `artifacts/outputs/r3d/rlbench_smokes/launch_smoke_open_drawer.stderr`
+## Generated Datasets
 
-Raw values from the current `open_drawer` launch smoke:
+- `artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt`
+- `artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt`
 
-| Field | Value |
-| --- | --- |
-| task | `RightOpenDrawer` |
-| headless | `true` |
-| front_rgb_shape | `[224, 224, 3]` |
-| wrist_left_rgb_shape | `[224, 224, 3]` |
-| wrist_right_rgb_shape | `[224, 224, 3]` |
-| action_finite | `true` |
-| action_dim | `18` |
-| reward | `0.0` |
-| done | `false` |
+## Generated Checkpoints
 
-### Open-Drawer Rollout
+- `artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/`
+- `artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/`
+- `artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/`
+- `artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/`
 
-File:
+## Raw Result Summary
 
-- `artifacts/outputs/r3d/rlbench_open_drawer_r3d_rollout/rollout_eval.json`
+### Proxy Serious Comparisons
 
-Raw values:
-
-| Field | Value |
-| --- | --- |
-| plan_requested | `true` |
-| plan_applied | `true` |
-| task_class | `RightOpenDrawer` |
-| episodes_per_task | `1` |
-| episode_length | `5` |
-| mean_success | `0.0` |
-| mean_return | `0.0` |
-| reset_retries | `[0]` |
-
-## Raw PerAct2 Integration Output
+| File | Reference mean success | Compared mean success |
+| --- | ---: | ---: |
+| `results/2026-03-25-runpod/reports/reveal_handoff_compare_serious/reveal_benchmark.json` | 0.583333 | 0.216667 |
+| `results/2026-03-25-runpod/reports/reveal_handoff_compare_serious_compact/reveal_benchmark.json` | 0.583333 | 0.520000 |
+| `results/2026-03-25-runpod/reports/reveal_phase_compare_serious_compact/reveal_benchmark.json` | 0.583333 | 0.513333 |
+| `results/2026-03-25-runpod/reports/reveal_phase_compare_serious_spatial_compactwm/reveal_benchmark.json` | 0.583333 | 0.493333 |
 
-Files:
+### Proxy Ablations
 
-- `artifacts/outputs/r3d/peract2_13_launch_smoke/launch_smoke_summary.json`
-- `artifacts/outputs/r3d/peract2_13_launch_smoke/launch_smoke_summary.md`
+- Full ablation matrix:
+  - `results/2026-03-25-runpod/reports/reveal_phase_ablations_compact/ablations.json`
+- Teacher audit:
+  - `results/2026-03-25-runpod/reports/reveal_teacher_audit_serious/teacher_audit.json`
 
-Raw values:
+### RLBench
 
-| Field | Value |
-| --- | --- |
-| task_count | `13` |
-| launch_successes | `13` |
-| finite_action_tasks | `13` |
-| error_tasks | `[]` |
-| resolution | `224` |
-| headless | `true` |
+| File | Mean success |
+| --- | ---: |
+| `results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/rollout_eval.json` | 0.000000 |
+| `results/2026-03-25-runpod/reports/peract2_spatial_full_ep1/spatial_phase_seed17_noplan_split/rollout_eval.json` | 0.000000 |
+| `results/2026-03-25-runpod/reports/peract2_spatial_full_ep1/spatial_phase_seed17_plan_split/rollout_eval.json` | 0.000000 |
 
-Per-task stdout/stderr/command logs are stored under:
+## Detailed Raw Index
 
-- `artifacts/outputs/r3d/peract2_13_launch_smoke/`
+- `results/2026-03-25-runpod/README.md`
 
-## Additional Indexes
+## Environment Recreation
 
-- Historical baseline note:
-  - `regression/baselines.md`
-- Phase-by-phase gate tracking:
-  - `results/phase_tracking.md`
-- Checkpoint and artifact index:
-  - `MODEL_INDEX.md`
+- `environment/README.md`
+- `environment/setup_same_machine.sh`
+- `environment/validate_same_machine.sh`
+- `environment/runtime_env_vars.sh`
+- `environment/upstream_revisions.txt`
+- `environment/rlbench_env_export.yaml`
+- `environment/rlbench_env_explicit.txt`
+- `environment/rlbench_pip_freeze.txt`

artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:918679191157abb31b3523be4b69ff7b95da4c373d130dd24a0db1314b57ec19
+size 583377508

artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1b1b90c882067eba271f59430b783c7fda2edbe6f221360a245ef32eef602d1
+size 200844508

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/checkpoint_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:360037583572613590e99f5f06766729edbecfc4ee2ea0950fe12bb08e562d83
+size 940662478

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/config_resolved.yaml

@@ -0,0 +1,153 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17
+output_dir: /workspace/outputs/r3d_handoff
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies:
+  - foliage_proxy
+  - bag_proxy
+  - cloth_proxy
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 3
+  batch_size: 4
+  num_workers: 24
+  lr: 0.0001
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: compact_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 1.0
+  phase: 0.05
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.12
+  persistence: 0.06
+  disturbance: 0.06
+  world_model: 0.2
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.06
+  support_stability: 0.06
+  reocclusion: 0.06
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.2
+  planner_risk: 0.08
+  planner_ranking: 0.2
+  proposal_reconstruction: 0.08
+  proposal_success: 0.12
+  proposal_ranking: 0.15
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.05

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/metrics.json

@@ -0,0 +1,179 @@
+[
+  {
+    "epoch": 0,
+    "train": {
+      "action": 0.01788575194874092,
+      "arm_role": 0.001019889743704545,
+      "belief": 0.12027505389169643,
+      "clearance": 0.08003731069988326,
+      "corridor": 0.2066852554580883,
+      "disturbance": 0.002048209719176061,
+      "grasp_affordance": 0.009688427269850907,
+      "occluder_contact": 0.21002381004785237,
+      "persistence": 0.6925194860485039,
+      "phase": 0.2873851528293208,
+      "planner_ranking": 0.007970526718954237,
+      "planner_risk": 0.022958766870004567,
+      "planner_success": 0.11003280751603214,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.3698378854676297,
+      "proposal_reconstruction": 0.10649478184549432,
+      "proposal_success": 0.3088213802952515,
+      "reocclusion": 0.23717812233065305,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.024943622789884868,
+      "support_stability": 0.12860428792865652,
+      "task_metrics": 0.16847629100084305,
+      "total": 0.7690052716355574,
+      "uncertainty": 8.382000443433706e-05,
+      "visibility": 0.11110214183205053,
+      "world_model": 2.4172904924342506
+    },
+    "val": {
+      "action": 0.01380122694271532,
+      "arm_role": 0.0007760834101425258,
+      "belief": 0.10585840746308818,
+      "clearance": 0.07610336713718646,
+      "corridor": 0.20833940104101645,
+      "disturbance": 0.001970026997503627,
+      "grasp_affordance": 0.009207394397394224,
+      "occluder_contact": 0.20593324529402185,
+      "persistence": 0.9972314130176197,
+      "phase": 0.32413111886743345,
+      "planner_ranking": 0.00022911480162733687,
+      "planner_risk": 0.01705723936020425,
+      "planner_success": 0.01417768012845155,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.148933302498225,
+      "proposal_reconstruction": 0.08428755696072723,
+      "proposal_success": 0.1355827044356953,
+      "reocclusion": 0.3386235964117628,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.00921160001025507,
+      "support_stability": 0.13572556762532753,
+      "task_metrics": 0.14731343361464413,
+      "total": 0.5933836915276267,
+      "uncertainty": 6.096601287991331e-05,
+      "visibility": 0.0983928834850138,
+      "world_model": 1.8323120962489734
+    }
+  },
+  {
+    "epoch": 1,
+    "train": {
+      "action": 0.011145903746058282,
+      "arm_role": 0.004716875678614566,
+      "belief": 0.10514058776591953,
+      "clearance": 0.07779615305756268,
+      "corridor": 0.20047297412529588,
+      "disturbance": 0.0022256996764458323,
+      "grasp_affordance": 0.009927417171236717,
+      "occluder_contact": 0.20330693313949985,
+      "persistence": 0.6114087605882608,
+      "phase": 0.21543656336633782,
+      "planner_ranking": 0.00018106158740920363,
+      "planner_risk": 0.015639749421787107,
+      "planner_success": 0.007014186491601561,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.08341658792801593,
+      "proposal_reconstruction": 0.07808616882876346,
+      "proposal_success": 0.08362671854464632,
+      "reocclusion": 0.21602793348659027,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.002888570647490652,
+      "support_stability": 0.1253421003685186,
+      "task_metrics": 0.14696427873874965,
+      "total": 0.4685811519622803,
+      "uncertainty": 3.833678546901578e-05,
+      "visibility": 0.09538611636350029,
+      "world_model": 1.5017830422050074
+    },
+    "val": {
+      "action": 0.011654359860007058,
+      "arm_role": 0.0034928608396457453,
+      "belief": 0.09692509336905046,
+      "clearance": 0.07511017166755417,
+      "corridor": 0.19370697032321582,
+      "disturbance": 0.0026899561648447575,
+      "grasp_affordance": 0.0108991796574132,
+      "occluder_contact": 0.20079099861058322,
+      "persistence": 0.7645651453146429,
+      "phase": 0.3479848448751551,
+      "planner_ranking": 7.394229859611104e-05,
+      "planner_risk": 0.015624796357852492,
+      "planner_success": 0.004826839748685333,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.10358103387283557,
+      "proposal_reconstruction": 0.0756011808460409,
+      "proposal_success": 0.07432993885242578,
+      "reocclusion": 0.2191494649106806,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0027684949190271172,
+      "support_stability": 0.1332334725919998,
+      "task_metrics": 0.1422290007273356,
+      "total": 0.4283526196624293,
+      "uncertainty": 6.312012101950654e-05,
+      "visibility": 0.09116258178696487,
+      "world_model": 1.2195359631018206
+    }
+  },
+  {
+    "epoch": 2,
+    "train": {
+      "action": 0.009001491786176829,
+      "arm_role": 0.0031435395541944003,
+      "belief": 0.09910694141136972,
+      "clearance": 0.0763780973459545,
+      "corridor": 0.1971655124798417,
+      "disturbance": 0.0022866554429607565,
+      "grasp_affordance": 0.010611724677054506,
+      "occluder_contact": 0.19787568214692566,
+      "persistence": 0.6391040890734171,
+      "phase": 0.2374798740211286,
+      "planner_ranking": 0.00014213861397461427,
+      "planner_risk": 0.011339436628316579,
+      "planner_success": 0.002072299244249926,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06418653756478115,
+      "proposal_reconstruction": 0.07331816969733489,
+      "proposal_success": 0.04838304229472813,
+      "reocclusion": 0.21607661038931264,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0016624049136513158,
+      "support_stability": 0.12395783413789774,
+      "task_metrics": 0.14624216885943161,
+      "total": 0.38278010518927325,
+      "uncertainty": 2.0790473172083742e-05,
+      "visibility": 0.09234962655525458,
+      "world_model": 1.1216850475261086
+    },
+    "val": {
+      "action": 0.010441844330423257,
+      "arm_role": 0.0016049532427904055,
+      "belief": 0.09470506825230339,
+      "clearance": 0.07375384743014972,
+      "corridor": 0.19222540205175226,
+      "disturbance": 0.0019139318584083494,
+      "grasp_affordance": 0.010620580760366989,
+      "occluder_contact": 0.1952310868284919,
+      "persistence": 0.6828906978621627,
+      "phase": 0.2299347263901974,
+      "planner_ranking": 0.00015113159439682602,
+      "planner_risk": 0.008906871584864954,
+      "planner_success": 0.0014728186387484047,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.08296609080086152,
+      "proposal_reconstruction": 0.07454461446314146,
+      "proposal_success": 0.05527880562074257,
+      "reocclusion": 0.21840402348475021,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0009884886958664565,
+      "support_stability": 0.13373579051006923,
+      "task_metrics": 0.14126789863362457,
+      "total": 0.3722763684662906,
+      "uncertainty": 3.5526475199510585e-05,
+      "visibility": 0.08942856323538405,
+      "world_model": 1.038636032379035
+    }
+  }
+]

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/summary.json

@@ -0,0 +1,103 @@
+{
+  "experiment_name": "proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17",
+  "device": "cuda",
+  "best_checkpoint": "/workspace/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/checkpoint_best.pt",
+  "final_train_total": 0.38278010518927325,
+  "final_val_total": 0.3722763684662906,
+  "train_time_sec": 108.92269134521484,
+  "peak_gpu_memory_mb": 2451.3857421875,
+  "num_train_samples": 380,
+  "num_val_samples": 131,
+  "planner_mode": "trainable",
+  "frozen_modules": [],
+  "init_info": {
+    "path": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+    "loaded_keys": 828,
+    "skipped_shape_mismatch_keys": [
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight"
+    ],
+    "missing_keys": [
+      "decoder.task_embedding.weight",
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight",
+      "decoder.mode_residual_heads.6.0.weight",
+      "decoder.mode_residual_heads.6.0.bias",
+      "decoder.mode_residual_heads.6.1.weight",
+      "decoder.mode_residual_heads.6.1.bias",
+      "decoder.mode_residual_heads.6.3.weight",
+      "decoder.mode_residual_heads.6.3.bias",
+      "elastic_state_head.decoder.task_embedding.weight",
+      "elastic_state_head.decoder.task_field_affine.weight",
+      "elastic_state_head.decoder.task_field_affine.bias",
+      "elastic_state_head.decoder.task_summary_adapter.0.weight",
+      "elastic_state_head.decoder.task_summary_adapter.0.bias",
+      "elastic_state_head.decoder.task_summary_adapter.1.weight",
+      "elastic_state_head.decoder.task_summary_adapter.1.bias",
+      "elastic_state_head.decoder.task_phase_head.weight",
+      "elastic_state_head.decoder.task_phase_head.bias",
+      "elastic_state_head.decoder.task_support_head.weight",
+      "elastic_state_head.decoder.task_support_head.bias",
+      "elastic_state_head.decoder.task_reocclusion_head.weight",
+      "elastic_state_head.decoder.task_reocclusion_head.bias",
+      "elastic_state_head.decoder.task_metric_head.0.weight",
+      "elastic_state_head.decoder.task_metric_head.0.bias",
+      "elastic_state_head.decoder.task_metric_head.1.weight",
+      "elastic_state_head.decoder.task_metric_head.1.bias",
+      "elastic_state_head.decoder.task_metric_head.3.weight",
+      "elastic_state_head.decoder.task_metric_head.3.bias",
+      "world_model.task_embedding.weight",
+      "world_model.spatial_field_encoder.0.weight",
+      "world_model.spatial_field_encoder.0.bias",
+      "world_model.spatial_field_encoder.2.weight",
+      "world_model.spatial_field_encoder.2.bias",
+      "world_model.spatial_context_proj.0.weight",
+      "world_model.spatial_context_proj.0.bias",
+      "world_model.spatial_context_proj.1.weight",
+      "world_model.spatial_context_proj.1.bias",
+      "world_model.spatial_gate_z.weight",
+      "world_model.spatial_gate_z.bias",
+      "world_model.spatial_gate_r.weight",
+      "world_model.spatial_gate_r.bias",
+      "world_model.spatial_candidate.weight",
+      "world_model.spatial_candidate.bias",
+      "world_model.spatial_summary_proj.0.weight",
+      "world_model.spatial_summary_proj.0.bias",
+      "world_model.spatial_summary_proj.1.weight",
+      "world_model.spatial_summary_proj.1.bias",
+      "world_model.spatial_phase_head.weight",
+      "world_model.spatial_phase_head.bias",
+      "world_model.spatial_support_mode_head.weight",
+      "world_model.spatial_support_mode_head.bias",
+      "world_model.spatial_arm_role_head.weight",
+      "world_model.spatial_arm_role_head.bias",
+      "world_model.spatial_reocclusion_head.weight",
+      "world_model.spatial_reocclusion_head.bias",
+      "world_model.spatial_target_belief_head.weight",
+      "world_model.spatial_target_belief_head.bias",
+      "world_model.spatial_visibility_head.weight",
+      "world_model.spatial_visibility_head.bias",
+      "world_model.spatial_clearance_head.weight",
+      "world_model.spatial_clearance_head.bias",
+      "world_model.spatial_occluder_contact_head.weight",
+      "world_model.spatial_occluder_contact_head.bias",
+      "world_model.spatial_grasp_affordance_head.weight",
+      "world_model.spatial_grasp_affordance_head.bias",
+      "world_model.spatial_support_stability_head.weight",
+      "world_model.spatial_support_stability_head.bias",
+      "world_model.spatial_persistence_head.weight",
+      "world_model.spatial_persistence_head.bias",
+      "world_model.spatial_reocclusion_field_head.weight",
+      "world_model.spatial_reocclusion_field_head.bias",
+      "world_model.spatial_disturbance_head.weight",
+      "world_model.spatial_disturbance_head.bias",
+      "world_model.spatial_uncertainty_head.weight",
+      "world_model.spatial_uncertainty_head.bias",
+      "world_model.spatial_access_head.weight",
+      "world_model.spatial_access_head.bias"
+    ],
+    "unexpected_keys": []
+  }
+}

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/checkpoint_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1bf52506b13c794b6d8f6f4738294947703c16c6a2c3b46dc8ea68fd14e0c12
+size 940663118

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/config_resolved.yaml

@@ -0,0 +1,153 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17
+output_dir: /workspace/outputs/r3d_handoff
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies:
+  - foliage_proxy
+  - bag_proxy
+  - cloth_proxy
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 5
+  batch_size: 4
+  num_workers: 24
+  lr: 0.00015
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: spatial_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 0.6
+  phase: 0.05
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.15
+  persistence: 0.08
+  disturbance: 0.08
+  world_model: 0.35
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.08
+  support_stability: 0.08
+  reocclusion: 0.08
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.25
+  planner_risk: 0.1
+  planner_ranking: 0.25
+  proposal_reconstruction: 0.05
+  proposal_success: 0.2
+  proposal_ranking: 0.25
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.1

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/metrics.json

@@ -0,0 +1,297 @@
+[
+  {
+    "epoch": 0,
+    "train": {
+      "action": 0.01881810994328637,
+      "arm_role": 2.660249408922697e-07,
+      "belief": 0.11447437676159959,
+      "clearance": 0.08659498787632114,
+      "corridor": 0.2168508012632006,
+      "disturbance": 0.003011604138699017,
+      "grasp_affordance": 0.026647591468338904,
+      "occluder_contact": 0.23407171917589087,
+      "persistence": 0.715272118206332,
+      "phase": 0.3055733912869504,
+      "planner_ranking": 0.022415388478637062,
+      "planner_risk": 0.023148000005044436,
+      "planner_success": 0.10891202024527286,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.29254391558076204,
+      "proposal_reconstruction": 0.10000501573085785,
+      "proposal_success": 0.24436878922738528,
+      "reocclusion": 0.23760041894489212,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.029171819122214067,
+      "support_stability": 0.13612447682964174,
+      "task_metrics": 0.15792004442528673,
+      "total": 1.1042629627805007,
+      "uncertainty": 0.0002569015418885101,
+      "visibility": 0.11963542550802231,
+      "world_model": 2.1293361720285917
+    },
+    "val": {
+      "action": 0.01775987928902561,
+      "arm_role": 1.9868213740892315e-08,
+      "belief": 0.1041581260435509,
+      "clearance": 0.07728264966245854,
+      "corridor": 0.2031804034204194,
+      "disturbance": 0.0017973093819102469,
+      "grasp_affordance": 0.029909261431770796,
+      "occluder_contact": 0.23528439167774085,
+      "persistence": 0.775493811025764,
+      "phase": 0.3234691350636157,
+      "planner_ranking": 0.0003446909185647724,
+      "planner_risk": 0.01719488257147146,
+      "planner_success": 0.00382949538867582,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.12049920004651402,
+      "proposal_reconstruction": 0.07953478553981493,
+      "proposal_success": 0.08874417897878271,
+      "reocclusion": 0.2280347410476569,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0012850317253844078,
+      "support_stability": 0.14258646694096652,
+      "task_metrics": 0.14477597267338724,
+      "total": 0.5612193951101014,
+      "uncertainty": 5.167457964654948e-05,
+      "visibility": 0.09895570508458397,
+      "world_model": 0.8950341885740106
+    }
+  },
+  {
+    "epoch": 1,
+    "train": {
+      "action": 0.011799810049859317,
+      "arm_role": 2.227331462659334e-08,
+      "belief": 0.10664581923108352,
+      "clearance": 0.08073694133444836,
+      "corridor": 0.20818061490886305,
+      "disturbance": 0.0034403698920198763,
+      "grasp_affordance": 0.02138785579682965,
+      "occluder_contact": 0.22385836949473933,
+      "persistence": 0.6393873089823358,
+      "phase": 0.23239254700510126,
+      "planner_ranking": 7.025458191932151e-05,
+      "planner_risk": 0.014862261340022087,
+      "planner_success": 0.0018582946357415303,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.07140745759304416,
+      "proposal_reconstruction": 0.07342678826106222,
+      "proposal_success": 0.052891033456513754,
+      "reocclusion": 0.22496669194415997,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0026235160074735944,
+      "support_stability": 0.129776736858644,
+      "task_metrics": 0.14390431940555573,
+      "total": 0.5859858243088973,
+      "uncertainty": 4.795166801757564e-05,
+      "visibility": 0.0989507636741588,
+      "world_model": 1.0815125898311013
+    },
+    "val": {
+      "action": 0.012531062933813893,
+      "arm_role": 0.0,
+      "belief": 0.1047917457692551,
+      "clearance": 0.08090435149091663,
+      "corridor": 0.20609694809624643,
+      "disturbance": 0.004269244044487344,
+      "grasp_affordance": 0.03268951613625342,
+      "occluder_contact": 0.2295533585729021,
+      "persistence": 1.1918026357889175,
+      "phase": 0.28549350922306377,
+      "planner_ranking": 6.612739407914474e-05,
+      "planner_risk": 0.008759455501355908,
+      "planner_success": 0.00080455597895762,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.1058380516866843,
+      "proposal_reconstruction": 0.07328326593745839,
+      "proposal_success": 0.058020667765628205,
+      "reocclusion": 0.33030271168911096,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.007980566662312909,
+      "support_stability": 0.13863918806115785,
+      "task_metrics": 0.14007962495088577,
+      "total": 0.5981471421140613,
+      "uncertainty": 4.717415575242106e-05,
+      "visibility": 0.09760378188256061,
+      "world_model": 0.9283028335282297
+    }
+  },
+  {
+    "epoch": 2,
+    "train": {
+      "action": 0.01041558725563319,
+      "arm_role": 8.783842387952302e-09,
+      "belief": 0.1059942942700888,
+      "clearance": 0.08004858184017632,
+      "corridor": 0.20489231364703492,
+      "disturbance": 0.0035354765677383464,
+      "grasp_affordance": 0.0182398099795376,
+      "occluder_contact": 0.2177388886087819,
+      "persistence": 0.7130741648352629,
+      "phase": 0.2232393227125469,
+      "planner_ranking": 9.544059870988445e-05,
+      "planner_risk": 0.007491795662300367,
+      "planner_success": 0.0006995215439115111,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06466802976731408,
+      "proposal_reconstruction": 0.07109034053589168,
+      "proposal_success": 0.03924152674643617,
+      "reocclusion": 0.24476394773840807,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0035586046545129073,
+      "support_stability": 0.1275901448373732,
+      "task_metrics": 0.14374801657701794,
+      "total": 0.5572637532886706,
+      "uncertainty": 3.1927780274169266e-05,
+      "visibility": 0.09604649849628147,
+      "world_model": 1.0012797054491545
+    },
+    "val": {
+      "action": 0.01250085191603637,
+      "arm_role": 1.9868213740892315e-08,
+      "belief": 0.10097009620883247,
+      "clearance": 0.07564825914574391,
+      "corridor": 0.19414961970213687,
+      "disturbance": 0.0028900962097946886,
+      "grasp_affordance": 0.014792599320185907,
+      "occluder_contact": 0.21100006365414822,
+      "persistence": 0.7430237780014673,
+      "phase": 0.21708492669418003,
+      "planner_ranking": 4.6006352427026066e-05,
+      "planner_risk": 0.004453302675039705,
+      "planner_success": 0.0002918489108472413,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.0847416734464015,
+      "proposal_reconstruction": 0.07325861490134036,
+      "proposal_success": 0.04800050914513342,
+      "reocclusion": 0.21185640358563626,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0001279175175472064,
+      "support_stability": 0.1355795610808965,
+      "task_metrics": 0.13995845525553732,
+      "total": 0.5864624706181613,
+      "uncertainty": 3.328838372660605e-05,
+      "visibility": 0.0915211413168546,
+      "world_model": 1.0730400555061572
+    }
+  },
+  {
+    "epoch": 3,
+    "train": {
+      "action": 0.009735075987287258,
+      "arm_role": 6.901590447676809e-09,
+      "belief": 0.10281775146722794,
+      "clearance": 0.07848918745784383,
+      "corridor": 0.20201588633813355,
+      "disturbance": 0.0031908015084091425,
+      "grasp_affordance": 0.015698057898369276,
+      "occluder_contact": 0.2077378544368242,
+      "persistence": 0.6618055252505368,
+      "phase": 0.2315950472103922,
+      "planner_ranking": 4.5244067302076614e-05,
+      "planner_risk": 0.004139781545381993,
+      "planner_success": 0.00024409074271955575,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.054642489266361255,
+      "proposal_reconstruction": 0.0706897854020721,
+      "proposal_success": 0.02984069204448085,
+      "reocclusion": 0.21985463225527813,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.00018626294637981214,
+      "support_stability": 0.12553868676094632,
+      "task_metrics": 0.14241597934773093,
+      "total": 0.5318458660652763,
+      "uncertainty": 2.4771544117715837e-05,
+      "visibility": 0.09392199994702088,
+      "world_model": 0.9659816164719431
+    },
+    "val": {
+      "action": 0.010274204266765579,
+      "arm_role": 0.0,
+      "belief": 0.09872564831466386,
+      "clearance": 0.07471577507076842,
+      "corridor": 0.19408509180401312,
+      "disturbance": 0.00259435343862909,
+      "grasp_affordance": 0.012839911272749305,
+      "occluder_contact": 0.20571506023406982,
+      "persistence": 0.7026729972073527,
+      "phase": 0.26675827372254746,
+      "planner_ranking": 4.497824862124672e-05,
+      "planner_risk": 0.002781865681754425,
+      "planner_success": 0.00015440414228413084,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.07663222694783613,
+      "proposal_reconstruction": 0.07091993872414935,
+      "proposal_success": 0.03695757263763384,
+      "reocclusion": 0.21823022356539062,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.00015865708998965354,
+      "support_stability": 0.13456520460771793,
+      "task_metrics": 0.1404704556546428,
+      "total": 0.5019133334810083,
+      "uncertainty": 1.7504166497460435e-05,
+      "visibility": 0.09078080579638481,
+      "world_model": 0.8508174094286832
+    }
+  },
+  {
+    "epoch": 4,
+    "train": {
+      "action": 0.00924236060757386,
+      "arm_role": 3.1370865671258223e-10,
+      "belief": 0.10063675104787476,
+      "clearance": 0.07763459076614757,
+      "corridor": 0.1999763826496507,
+      "disturbance": 0.0032564817505006337,
+      "grasp_affordance": 0.015768864574401003,
+      "occluder_contact": 0.20453226503572966,
+      "persistence": 0.6381541584980656,
+      "phase": 0.23467233871158802,
+      "planner_ranking": 0.002148842357724178,
+      "planner_risk": 0.005933802830986679,
+      "planner_success": 0.0012002266089487085,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.04519814905391908,
+      "proposal_reconstruction": 0.07035028267847865,
+      "proposal_success": 0.02132791725330447,
+      "reocclusion": 0.21220772236181226,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.00017794872585095856,
+      "support_stability": 0.12488894654732001,
+      "task_metrics": 0.14187381208727234,
+      "total": 0.5180026358679721,
+      "uncertainty": 1.678193236126677e-05,
+      "visibility": 0.09251636654922837,
+      "world_model": 0.9452698754636865
+    },
+    "val": {
+      "action": 0.009614509682058158,
+      "arm_role": 9.934106870446158e-09,
+      "belief": 0.0954468369935498,
+      "clearance": 0.07359647931474628,
+      "corridor": 0.19544327168753653,
+      "disturbance": 0.004919912796388168,
+      "grasp_affordance": 0.01787316725786888,
+      "occluder_contact": 0.2034355541973403,
+      "persistence": 0.8611267923631452,
+      "phase": 0.2359058087635221,
+      "planner_ranking": 7.647072767337125e-06,
+      "planner_risk": 0.0028961390207493396,
+      "planner_success": 0.00021185575601658925,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06654375551398985,
+      "proposal_reconstruction": 0.07095728814601898,
+      "proposal_success": 0.02652511727347067,
+      "reocclusion": 0.28424677580143465,
+      "role_swap_consistency": 0.0,
+      "support_mode": 5.328719337491996e-05,
+      "support_stability": 0.13284552616603446,
+      "task_metrics": 0.1393586558600267,
+      "total": 0.5038685834769047,
+      "uncertainty": 1.3334046157483085e-05,
+      "visibility": 0.08824686581889789,
+      "world_model": 0.8240700237678759
+    }
+  }
+]

artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/summary.json

@@ -0,0 +1,103 @@
+{
+  "experiment_name": "proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17",
+  "device": "cuda",
+  "best_checkpoint": "/workspace/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/checkpoint_best.pt",
+  "final_train_total": 0.5180026358679721,
+  "final_val_total": 0.5038685834769047,
+  "train_time_sec": 163.31340551376343,
+  "peak_gpu_memory_mb": 2924.82177734375,
+  "num_train_samples": 380,
+  "num_val_samples": 131,
+  "planner_mode": "trainable",
+  "frozen_modules": [],
+  "init_info": {
+    "path": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+    "loaded_keys": 828,
+    "skipped_shape_mismatch_keys": [
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight"
+    ],
+    "missing_keys": [
+      "decoder.task_embedding.weight",
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight",
+      "decoder.mode_residual_heads.6.0.weight",
+      "decoder.mode_residual_heads.6.0.bias",
+      "decoder.mode_residual_heads.6.1.weight",
+      "decoder.mode_residual_heads.6.1.bias",
+      "decoder.mode_residual_heads.6.3.weight",
+      "decoder.mode_residual_heads.6.3.bias",
+      "elastic_state_head.decoder.task_embedding.weight",
+      "elastic_state_head.decoder.task_field_affine.weight",
+      "elastic_state_head.decoder.task_field_affine.bias",
+      "elastic_state_head.decoder.task_summary_adapter.0.weight",
+      "elastic_state_head.decoder.task_summary_adapter.0.bias",
+      "elastic_state_head.decoder.task_summary_adapter.1.weight",
+      "elastic_state_head.decoder.task_summary_adapter.1.bias",
+      "elastic_state_head.decoder.task_phase_head.weight",
+      "elastic_state_head.decoder.task_phase_head.bias",
+      "elastic_state_head.decoder.task_support_head.weight",
+      "elastic_state_head.decoder.task_support_head.bias",
+      "elastic_state_head.decoder.task_reocclusion_head.weight",
+      "elastic_state_head.decoder.task_reocclusion_head.bias",
+      "elastic_state_head.decoder.task_metric_head.0.weight",
+      "elastic_state_head.decoder.task_metric_head.0.bias",
+      "elastic_state_head.decoder.task_metric_head.1.weight",
+      "elastic_state_head.decoder.task_metric_head.1.bias",
+      "elastic_state_head.decoder.task_metric_head.3.weight",
+      "elastic_state_head.decoder.task_metric_head.3.bias",
+      "world_model.task_embedding.weight",
+      "world_model.spatial_field_encoder.0.weight",
+      "world_model.spatial_field_encoder.0.bias",
+      "world_model.spatial_field_encoder.2.weight",
+      "world_model.spatial_field_encoder.2.bias",
+      "world_model.spatial_context_proj.0.weight",
+      "world_model.spatial_context_proj.0.bias",
+      "world_model.spatial_context_proj.1.weight",
+      "world_model.spatial_context_proj.1.bias",
+      "world_model.spatial_gate_z.weight",
+      "world_model.spatial_gate_z.bias",
+      "world_model.spatial_gate_r.weight",
+      "world_model.spatial_gate_r.bias",
+      "world_model.spatial_candidate.weight",
+      "world_model.spatial_candidate.bias",
+      "world_model.spatial_summary_proj.0.weight",
+      "world_model.spatial_summary_proj.0.bias",
+      "world_model.spatial_summary_proj.1.weight",
+      "world_model.spatial_summary_proj.1.bias",
+      "world_model.spatial_phase_head.weight",
+      "world_model.spatial_phase_head.bias",
+      "world_model.spatial_support_mode_head.weight",
+      "world_model.spatial_support_mode_head.bias",
+      "world_model.spatial_arm_role_head.weight",
+      "world_model.spatial_arm_role_head.bias",
+      "world_model.spatial_reocclusion_head.weight",
+      "world_model.spatial_reocclusion_head.bias",
+      "world_model.spatial_target_belief_head.weight",
+      "world_model.spatial_target_belief_head.bias",
+      "world_model.spatial_visibility_head.weight",
+      "world_model.spatial_visibility_head.bias",
+      "world_model.spatial_clearance_head.weight",
+      "world_model.spatial_clearance_head.bias",
+      "world_model.spatial_occluder_contact_head.weight",
+      "world_model.spatial_occluder_contact_head.bias",
+      "world_model.spatial_grasp_affordance_head.weight",
+      "world_model.spatial_grasp_affordance_head.bias",
+      "world_model.spatial_support_stability_head.weight",
+      "world_model.spatial_support_stability_head.bias",
+      "world_model.spatial_persistence_head.weight",
+      "world_model.spatial_persistence_head.bias",
+      "world_model.spatial_reocclusion_field_head.weight",
+      "world_model.spatial_reocclusion_field_head.bias",
+      "world_model.spatial_disturbance_head.weight",
+      "world_model.spatial_disturbance_head.bias",
+      "world_model.spatial_uncertainty_head.weight",
+      "world_model.spatial_uncertainty_head.bias",
+      "world_model.spatial_access_head.weight",
+      "world_model.spatial_access_head.bias"
+    ],
+    "unexpected_keys": []
+  }
+}

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/checkpoint_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3968c7aaeace3aeeb9ba2a6343ab2b35b792acbd00911d5eb76d90cd3db80a1c
+size 940662478

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/config_resolved.yaml

@@ -0,0 +1,153 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17
+output_dir: /workspace/outputs/r3d_handoff_phase
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies:
+  - foliage_proxy
+  - bag_proxy
+  - cloth_proxy
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state_phase
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 3
+  batch_size: 4
+  num_workers: 24
+  lr: 0.0001
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: compact_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 1.0
+  phase: 0.08
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.12
+  persistence: 0.06
+  disturbance: 0.06
+  world_model: 0.2
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.06
+  support_stability: 0.06
+  reocclusion: 0.06
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.2
+  planner_risk: 0.08
+  planner_ranking: 0.2
+  proposal_reconstruction: 0.08
+  proposal_success: 0.12
+  proposal_ranking: 0.15
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.05

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/metrics.json

@@ -0,0 +1,179 @@
+[
+  {
+    "epoch": 0,
+    "train": {
+      "action": 0.01903669923838032,
+      "arm_role": 0.0011954489507173237,
+      "belief": 0.11834000094156516,
+      "clearance": 0.07986288266746622,
+      "corridor": 0.2066003169864416,
+      "disturbance": 0.00205025365499559,
+      "grasp_affordance": 0.00966440967382177,
+      "occluder_contact": 0.2094391022857867,
+      "persistence": 0.6896675200305732,
+      "phase": 0.3597658646734137,
+      "planner_ranking": 0.010135862800770586,
+      "planner_risk": 0.02269953653020294,
+      "planner_success": 0.10877308378878393,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.3664713734858914,
+      "proposal_reconstruction": 0.10571438976024326,
+      "proposal_success": 0.3060418816773515,
+      "reocclusion": 0.23476032413738337,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.02608335896542198,
+      "support_stability": 0.1283740213434947,
+      "task_metrics": 0.16838834607287456,
+      "total": 0.7838476789625067,
+      "uncertainty": 9.599007572480907e-05,
+      "visibility": 0.11086608359688206,
+      "world_model": 2.4192019870406702
+    },
+    "val": {
+      "action": 0.016467795622619717,
+      "arm_role": 0.00044356281741232976,
+      "belief": 0.10317085593035727,
+      "clearance": 0.07557447807806911,
+      "corridor": 0.20408162474632263,
+      "disturbance": 0.0014927912058986046,
+      "grasp_affordance": 0.00985990883782506,
+      "occluder_contact": 0.20460259372537787,
+      "persistence": 0.9340643006179369,
+      "phase": 0.4205810320422505,
+      "planner_ranking": 0.0002936208506016004,
+      "planner_risk": 0.0162254377748027,
+      "planner_success": 0.013315262752726223,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.14499353143301877,
+      "proposal_reconstruction": 0.08181179179386659,
+      "proposal_success": 0.13043119690634988,
+      "reocclusion": 0.2904955812475898,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0027611398766043058,
+      "support_stability": 0.13547231697223402,
+      "task_metrics": 0.1477968403787324,
+      "total": 0.6003618971867994,
+      "uncertainty": 0.00011726474721482109,
+      "visibility": 0.09890205435680621,
+      "world_model": 1.8151403015310115
+    }
+  },
+  {
+    "epoch": 1,
+    "train": {
+      "action": 0.011172730509976024,
+      "arm_role": 0.005047608049292314,
+      "belief": 0.10358183007491262,
+      "clearance": 0.07736518386947482,
+      "corridor": 0.19873161581589988,
+      "disturbance": 0.002192211224188979,
+      "grasp_affordance": 0.010328439559395376,
+      "occluder_contact": 0.20153354829863498,
+      "persistence": 0.6124827290352091,
+      "phase": 0.2595963217710194,
+      "planner_ranking": 0.00016108446396796545,
+      "planner_risk": 0.01525886312960402,
+      "planner_success": 0.006821451556721801,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.08119581977983839,
+      "proposal_reconstruction": 0.07775982775186238,
+      "proposal_success": 0.08087819657827679,
+      "reocclusion": 0.21806120462715625,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.004165554046630859,
+      "support_stability": 0.12518173395410964,
+      "task_metrics": 0.1470285534074432,
+      "total": 0.4728757431632594,
+      "uncertainty": 4.4979283201859e-05,
+      "visibility": 0.09486380823348697,
+      "world_model": 1.477295964642575
+    },
+    "val": {
+      "action": 0.014699659296170328,
+      "arm_role": 0.0032501910410990768,
+      "belief": 0.09604058076034892,
+      "clearance": 0.0737346127403505,
+      "corridor": 0.19246741181070154,
+      "disturbance": 0.002424581844631272,
+      "grasp_affordance": 0.011332590499836388,
+      "occluder_contact": 0.1972112443410989,
+      "persistence": 0.754733283637148,
+      "phase": 0.27422264163606364,
+      "planner_ranking": 6.61312957218439e-05,
+      "planner_risk": 0.014809876634513565,
+      "planner_success": 0.005034577334299684,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.10355714928697456,
+      "proposal_reconstruction": 0.07605304298075763,
+      "proposal_success": 0.07239359052795352,
+      "reocclusion": 0.2211838181723248,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0030512072068328657,
+      "support_stability": 0.13321079302466277,
+      "task_metrics": 0.14215884109338126,
+      "total": 0.42770718534787494,
+      "uncertainty": 4.0278254969604006e-05,
+      "visibility": 0.09008839387785304,
+      "world_model": 1.1843715039166538
+    }
+  },
+  {
+    "epoch": 2,
+    "train": {
+      "action": 0.009650313334637567,
+      "arm_role": 0.0029296059357492546,
+      "belief": 0.09819177420515764,
+      "clearance": 0.07597495360594046,
+      "corridor": 0.19736162976018692,
+      "disturbance": 0.002284994474719401,
+      "grasp_affordance": 0.011029037923895214,
+      "occluder_contact": 0.19601713622871197,
+      "persistence": 0.6342933682810613,
+      "phase": 0.2582773412528791,
+      "planner_ranking": 0.00024582118595916605,
+      "planner_risk": 0.011063184471506822,
+      "planner_success": 0.0019527532209299113,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06367454680761224,
+      "proposal_reconstruction": 0.07267745521507765,
+      "proposal_success": 0.047251041174719205,
+      "reocclusion": 0.2153781190392022,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0022934590515337494,
+      "support_stability": 0.12411213658357921,
+      "task_metrics": 0.14521450063115673,
+      "total": 0.3853033130106173,
+      "uncertainty": 2.1849837026509607e-05,
+      "visibility": 0.09172474040012611,
+      "world_model": 1.0909657170898035
+    },
+    "val": {
+      "action": 0.010615826838395813,
+      "arm_role": 0.002171783652474088,
+      "belief": 0.09422377767887982,
+      "clearance": 0.0732199704330979,
+      "corridor": 0.1925947627786434,
+      "disturbance": 0.001796035210700762,
+      "grasp_affordance": 0.010509083230951519,
+      "occluder_contact": 0.19385047753651938,
+      "persistence": 0.6774057000875473,
+      "phase": 0.35872898339188064,
+      "planner_ranking": 0.00021498550532586788,
+      "planner_risk": 0.009475005589510229,
+      "planner_success": 0.0013016004857931737,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.08124377544630658,
+      "proposal_reconstruction": 0.07418028581323045,
+      "proposal_success": 0.053315439402605545,
+      "reocclusion": 0.21817159472089825,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0010789622478610413,
+      "support_stability": 0.13355377282608638,
+      "task_metrics": 0.14122987561153644,
+      "total": 0.3789277257341327,
+      "uncertainty": 3.114792006079259e-05,
+      "visibility": 0.08864631684440555,
+      "world_model": 0.9896242853366968
+    }
+  }
+]

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/summary.json

@@ -0,0 +1,103 @@
+{
+  "experiment_name": "proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17",
+  "device": "cuda",
+  "best_checkpoint": "/workspace/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/checkpoint_best.pt",
+  "final_train_total": 0.3853033130106173,
+  "final_val_total": 0.3789277257341327,
+  "train_time_sec": 128.96558284759521,
+  "peak_gpu_memory_mb": 2450.287109375,
+  "num_train_samples": 380,
+  "num_val_samples": 131,
+  "planner_mode": "trainable",
+  "frozen_modules": [],
+  "init_info": {
+    "path": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+    "loaded_keys": 828,
+    "skipped_shape_mismatch_keys": [
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight"
+    ],
+    "missing_keys": [
+      "decoder.task_embedding.weight",
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight",
+      "decoder.mode_residual_heads.6.0.weight",
+      "decoder.mode_residual_heads.6.0.bias",
+      "decoder.mode_residual_heads.6.1.weight",
+      "decoder.mode_residual_heads.6.1.bias",
+      "decoder.mode_residual_heads.6.3.weight",
+      "decoder.mode_residual_heads.6.3.bias",
+      "elastic_state_head.decoder.task_embedding.weight",
+      "elastic_state_head.decoder.task_field_affine.weight",
+      "elastic_state_head.decoder.task_field_affine.bias",
+      "elastic_state_head.decoder.task_summary_adapter.0.weight",
+      "elastic_state_head.decoder.task_summary_adapter.0.bias",
+      "elastic_state_head.decoder.task_summary_adapter.1.weight",
+      "elastic_state_head.decoder.task_summary_adapter.1.bias",
+      "elastic_state_head.decoder.task_phase_head.weight",
+      "elastic_state_head.decoder.task_phase_head.bias",
+      "elastic_state_head.decoder.task_support_head.weight",
+      "elastic_state_head.decoder.task_support_head.bias",
+      "elastic_state_head.decoder.task_reocclusion_head.weight",
+      "elastic_state_head.decoder.task_reocclusion_head.bias",
+      "elastic_state_head.decoder.task_metric_head.0.weight",
+      "elastic_state_head.decoder.task_metric_head.0.bias",
+      "elastic_state_head.decoder.task_metric_head.1.weight",
+      "elastic_state_head.decoder.task_metric_head.1.bias",
+      "elastic_state_head.decoder.task_metric_head.3.weight",
+      "elastic_state_head.decoder.task_metric_head.3.bias",
+      "world_model.task_embedding.weight",
+      "world_model.spatial_field_encoder.0.weight",
+      "world_model.spatial_field_encoder.0.bias",
+      "world_model.spatial_field_encoder.2.weight",
+      "world_model.spatial_field_encoder.2.bias",
+      "world_model.spatial_context_proj.0.weight",
+      "world_model.spatial_context_proj.0.bias",
+      "world_model.spatial_context_proj.1.weight",
+      "world_model.spatial_context_proj.1.bias",
+      "world_model.spatial_gate_z.weight",
+      "world_model.spatial_gate_z.bias",
+      "world_model.spatial_gate_r.weight",
+      "world_model.spatial_gate_r.bias",
+      "world_model.spatial_candidate.weight",
+      "world_model.spatial_candidate.bias",
+      "world_model.spatial_summary_proj.0.weight",
+      "world_model.spatial_summary_proj.0.bias",
+      "world_model.spatial_summary_proj.1.weight",
+      "world_model.spatial_summary_proj.1.bias",
+      "world_model.spatial_phase_head.weight",
+      "world_model.spatial_phase_head.bias",
+      "world_model.spatial_support_mode_head.weight",
+      "world_model.spatial_support_mode_head.bias",
+      "world_model.spatial_arm_role_head.weight",
+      "world_model.spatial_arm_role_head.bias",
+      "world_model.spatial_reocclusion_head.weight",
+      "world_model.spatial_reocclusion_head.bias",
+      "world_model.spatial_target_belief_head.weight",
+      "world_model.spatial_target_belief_head.bias",
+      "world_model.spatial_visibility_head.weight",
+      "world_model.spatial_visibility_head.bias",
+      "world_model.spatial_clearance_head.weight",
+      "world_model.spatial_clearance_head.bias",
+      "world_model.spatial_occluder_contact_head.weight",
+      "world_model.spatial_occluder_contact_head.bias",
+      "world_model.spatial_grasp_affordance_head.weight",
+      "world_model.spatial_grasp_affordance_head.bias",
+      "world_model.spatial_support_stability_head.weight",
+      "world_model.spatial_support_stability_head.bias",
+      "world_model.spatial_persistence_head.weight",
+      "world_model.spatial_persistence_head.bias",
+      "world_model.spatial_reocclusion_field_head.weight",
+      "world_model.spatial_reocclusion_field_head.bias",
+      "world_model.spatial_disturbance_head.weight",
+      "world_model.spatial_disturbance_head.bias",
+      "world_model.spatial_uncertainty_head.weight",
+      "world_model.spatial_uncertainty_head.bias",
+      "world_model.spatial_access_head.weight",
+      "world_model.spatial_access_head.bias"
+    ],
+    "unexpected_keys": []
+  }
+}

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/checkpoint_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc40df2f2c241001bf2e8c17a177cfbcda82acef7ae90997d8e145357d901349
+size 940663118

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/config_resolved.yaml

@@ -0,0 +1,153 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17
+output_dir: /workspace/outputs/r3d_handoff_phase
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies:
+  - foliage_proxy
+  - bag_proxy
+  - cloth_proxy
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state_phase
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 4
+  batch_size: 4
+  num_workers: 24
+  lr: 0.00015
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: spatial_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 0.6
+  phase: 0.08
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.15
+  persistence: 0.08
+  disturbance: 0.08
+  world_model: 0.35
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.08
+  support_stability: 0.08
+  reocclusion: 0.08
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.25
+  planner_risk: 0.1
+  planner_ranking: 0.25
+  proposal_reconstruction: 0.05
+  proposal_success: 0.2
+  proposal_ranking: 0.25
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.1

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/metrics.json

@@ -0,0 +1,238 @@
+[
+  {
+    "epoch": 0,
+    "train": {
+      "action": 0.019181225144941555,
+      "arm_role": 3.1182640477230673e-07,
+      "belief": 0.11448187259467024,
+      "clearance": 0.0864047217133798,
+      "corridor": 0.21893725262856797,
+      "disturbance": 0.003079198535813607,
+      "grasp_affordance": 0.02621712978978298,
+      "occluder_contact": 0.2337783462122867,
+      "persistence": 0.719700569705425,
+      "phase": 0.3856928740677081,
+      "planner_ranking": 0.022916802166031102,
+      "planner_risk": 0.023334504154167678,
+      "planner_success": 0.10860285238411865,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.2922646708394352,
+      "proposal_reconstruction": 0.09971292505138799,
+      "proposal_success": 0.24467597556741613,
+      "reocclusion": 0.24054438805109576,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.03278662405515972,
+      "support_stability": 0.13613393926306774,
+      "task_metrics": 0.15838541047353494,
+      "total": 1.1187869517426743,
+      "uncertainty": 0.00028297689169975407,
+      "visibility": 0.11868352764531186,
+      "world_model": 2.122099159265819
+    },
+    "val": {
+      "action": 0.016976301219653,
+      "arm_role": 5.6895338003194025e-08,
+      "belief": 0.10437920215454968,
+      "clearance": 0.07773172505425685,
+      "corridor": 0.20573271156260461,
+      "disturbance": 0.002334372425979602,
+      "grasp_affordance": 0.04297696501298836,
+      "occluder_contact": 0.23979515650055624,
+      "persistence": 0.7753512069131389,
+      "phase": 0.44806069852502056,
+      "planner_ranking": 0.0006287269447208624,
+      "planner_risk": 0.018244338094849478,
+      "planner_success": 0.004400868512069185,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.12023697172602017,
+      "proposal_reconstruction": 0.07816453142599626,
+      "proposal_success": 0.08989288398262227,
+      "reocclusion": 0.23809522954803525,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.002169632747995131,
+      "support_stability": 0.1437560645016757,
+      "task_metrics": 0.14553586303284674,
+      "total": 0.6175428755355604,
+      "uncertainty": 7.00113979071444e-05,
+      "visibility": 0.09822999302184943,
+      "world_model": 0.9931504021991383
+    }
+  },
+  {
+    "epoch": 1,
+    "train": {
+      "action": 0.011922537655520597,
+      "arm_role": 3.544907820852179e-08,
+      "belief": 0.10706032749853636,
+      "clearance": 0.08102132846650324,
+      "corridor": 0.20774916157518564,
+      "disturbance": 0.003389558960426305,
+      "grasp_affordance": 0.021082493273149195,
+      "occluder_contact": 0.22566462457180023,
+      "persistence": 0.6525513453439712,
+      "phase": 0.2908584324937118,
+      "planner_ranking": 8.167305853045323e-05,
+      "planner_risk": 0.01491448436741178,
+      "planner_success": 0.0019715585316972513,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06992289091607458,
+      "proposal_reconstruction": 0.07328974894787135,
+      "proposal_success": 0.05622971397089331,
+      "reocclusion": 0.2235310155016027,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0016054740077570866,
+      "support_stability": 0.13015181959459657,
+      "task_metrics": 0.1444544498857699,
+      "total": 0.6000718728492135,
+      "uncertainty": 5.28337070471408e-05,
+      "visibility": 0.09962119711072821,
+      "world_model": 1.0844623948398389
+    },
+    "val": {
+      "action": 0.012654555594605026,
+      "arm_role": 0.0,
+      "belief": 0.10426627686529448,
+      "clearance": 0.08140931176868352,
+      "corridor": 0.21032546257430856,
+      "disturbance": 0.004191758795445132,
+      "grasp_affordance": 0.053641817603034506,
+      "occluder_contact": 0.23278299877137848,
+      "persistence": 1.2810496254400774,
+      "phase": 0.3091735607295325,
+      "planner_ranking": 0.00015768451346571743,
+      "planner_risk": 0.00882755185364548,
+      "planner_success": 0.0008127102125647732,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.1040246604490235,
+      "proposal_reconstruction": 0.074961857362227,
+      "proposal_success": 0.05817745603395231,
+      "reocclusion": 0.32169581723935675,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0020671126813712444,
+      "support_stability": 0.13979825738704565,
+      "task_metrics": 0.14051661414630484,
+      "total": 0.6931917875102072,
+      "uncertainty": 7.29499135531748e-05,
+      "visibility": 0.09667146341367201,
+      "world_model": 1.1483498287923408
+    }
+  },
+  {
+    "epoch": 2,
+    "train": {
+      "action": 0.009906618949025869,
+      "arm_role": 1.2548346268503289e-08,
+      "belief": 0.10543899598874544,
+      "clearance": 0.08008571463195902,
+      "corridor": 0.20356425615517718,
+      "disturbance": 0.003584925674122611,
+      "grasp_affordance": 0.02068159209662362,
+      "occluder_contact": 0.21795249863674765,
+      "persistence": 0.7022376383202515,
+      "phase": 0.2805413355952815,
+      "planner_ranking": 5.863887835115939e-05,
+      "planner_risk": 0.007802685193325344,
+      "planner_success": 0.0006161007331684232,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.059995323853371176,
+      "proposal_reconstruction": 0.0712106538446326,
+      "proposal_success": 0.03740891177011164,
+      "reocclusion": 0.23808821498992314,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0014756444253419575,
+      "support_stability": 0.1280404823783197,
+      "task_metrics": 0.14326360319790088,
+      "total": 0.5650381593327773,
+      "uncertainty": 4.478374925335964e-05,
+      "visibility": 0.09581161878610912,
+      "world_model": 1.0014131157021773
+    },
+    "val": {
+      "action": 0.011506305104403786,
+      "arm_role": 0.0,
+      "belief": 0.10174731707031076,
+      "clearance": 0.07599064165895636,
+      "corridor": 0.19603406366976825,
+      "disturbance": 0.0034094253584925987,
+      "grasp_affordance": 0.01412029577080499,
+      "occluder_contact": 0.21309178299976117,
+      "persistence": 0.7315149356921514,
+      "phase": 0.2930864508433098,
+      "planner_ranking": 0.0013956731370778168,
+      "planner_risk": 0.0041357744308753,
+      "planner_success": 0.0003894640280067864,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.08693639387263719,
+      "proposal_reconstruction": 0.07113255470088034,
+      "proposal_success": 0.05217952732786988,
+      "reocclusion": 0.2135303650390018,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0002804717289770699,
+      "support_stability": 0.13648639472596574,
+      "task_metrics": 0.13825015418908812,
+      "total": 0.621522765267979,
+      "uncertainty": 5.1016551851441335e-05,
+      "visibility": 0.0916972327412981,
+      "world_model": 1.1354843739307288
+    }
+  },
+  {
+    "epoch": 3,
+    "train": {
+      "action": 0.009076371156659565,
+      "arm_role": 3.764503880550987e-09,
+      "belief": 0.10279968522096936,
+      "clearance": 0.07841498155735041,
+      "corridor": 0.2006541040775023,
+      "disturbance": 0.00325027588436282,
+      "grasp_affordance": 0.014321190530532284,
+      "occluder_contact": 0.2105231849770797,
+      "persistence": 0.6405418016016483,
+      "phase": 0.24933996639753642,
+      "planner_ranking": 8.54384282053831e-05,
+      "planner_risk": 0.004359905376393152,
+      "planner_success": 0.00026648731834843365,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.048603357446968164,
+      "proposal_reconstruction": 0.07084132981927771,
+      "proposal_success": 0.02799873605958725,
+      "reocclusion": 0.2113562585295815,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0005042697253980135,
+      "support_stability": 0.1260564218226232,
+      "task_metrics": 0.14196890086719866,
+      "total": 0.5253663646547417,
+      "uncertainty": 2.7578330574023744e-05,
+      "visibility": 0.09414561410483561,
+      "world_model": 0.9370055976666902
+    },
+    "val": {
+      "action": 0.01043125390159813,
+      "arm_role": 0.0,
+      "belief": 0.09857180962959926,
+      "clearance": 0.07481752595666682,
+      "corridor": 0.19474056340528256,
+      "disturbance": 0.00304531856664019,
+      "grasp_affordance": 0.01476043919242467,
+      "occluder_contact": 0.20771399185513006,
+      "persistence": 0.6766496115561688,
+      "phase": 0.3030792711546052,
+      "planner_ranking": 6.782512608514011e-05,
+      "planner_risk": 0.002957202364500104,
+      "planner_success": 0.00018708011846297956,
+      "proposal_diversity": 0.0,
+      "proposal_ranking": 0.06918713425561278,
+      "proposal_reconstruction": 0.07121785075375528,
+      "proposal_success": 0.028793677348982204,
+      "reocclusion": 0.21873644242684046,
+      "role_swap_consistency": 0.0,
+      "support_mode": 0.0007097468169928161,
+      "support_stability": 0.13461551465319865,
+      "task_metrics": 0.13820900722886575,
+      "total": 0.5076249196673884,
+      "uncertainty": 2.9063129700675333e-05,
+      "visibility": 0.09054145137920525,
+      "world_model": 0.850949106794415
+    }
+  }
+]

artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/summary.json

@@ -0,0 +1,103 @@
+{
+  "experiment_name": "proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17",
+  "device": "cuda",
+  "best_checkpoint": "/workspace/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/checkpoint_best.pt",
+  "final_train_total": 0.5253663646547417,
+  "final_val_total": 0.5076249196673884,
+  "train_time_sec": 154.84144067764282,
+  "peak_gpu_memory_mb": 2926.07470703125,
+  "num_train_samples": 380,
+  "num_val_samples": 131,
+  "planner_mode": "trainable",
+  "frozen_modules": [],
+  "init_info": {
+    "path": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+    "loaded_keys": 828,
+    "skipped_shape_mismatch_keys": [
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight"
+    ],
+    "missing_keys": [
+      "decoder.task_embedding.weight",
+      "decoder.proposal_mode_head.3.weight",
+      "decoder.proposal_mode_head.3.bias",
+      "decoder.proposal_mode_embeddings.weight",
+      "decoder.mode_residual_heads.6.0.weight",
+      "decoder.mode_residual_heads.6.0.bias",
+      "decoder.mode_residual_heads.6.1.weight",
+      "decoder.mode_residual_heads.6.1.bias",
+      "decoder.mode_residual_heads.6.3.weight",
+      "decoder.mode_residual_heads.6.3.bias",
+      "elastic_state_head.decoder.task_embedding.weight",
+      "elastic_state_head.decoder.task_field_affine.weight",
+      "elastic_state_head.decoder.task_field_affine.bias",
+      "elastic_state_head.decoder.task_summary_adapter.0.weight",
+      "elastic_state_head.decoder.task_summary_adapter.0.bias",
+      "elastic_state_head.decoder.task_summary_adapter.1.weight",
+      "elastic_state_head.decoder.task_summary_adapter.1.bias",
+      "elastic_state_head.decoder.task_phase_head.weight",
+      "elastic_state_head.decoder.task_phase_head.bias",
+      "elastic_state_head.decoder.task_support_head.weight",
+      "elastic_state_head.decoder.task_support_head.bias",
+      "elastic_state_head.decoder.task_reocclusion_head.weight",
+      "elastic_state_head.decoder.task_reocclusion_head.bias",
+      "elastic_state_head.decoder.task_metric_head.0.weight",
+      "elastic_state_head.decoder.task_metric_head.0.bias",
+      "elastic_state_head.decoder.task_metric_head.1.weight",
+      "elastic_state_head.decoder.task_metric_head.1.bias",
+      "elastic_state_head.decoder.task_metric_head.3.weight",
+      "elastic_state_head.decoder.task_metric_head.3.bias",
+      "world_model.task_embedding.weight",
+      "world_model.spatial_field_encoder.0.weight",
+      "world_model.spatial_field_encoder.0.bias",
+      "world_model.spatial_field_encoder.2.weight",
+      "world_model.spatial_field_encoder.2.bias",
+      "world_model.spatial_context_proj.0.weight",
+      "world_model.spatial_context_proj.0.bias",
+      "world_model.spatial_context_proj.1.weight",
+      "world_model.spatial_context_proj.1.bias",
+      "world_model.spatial_gate_z.weight",
+      "world_model.spatial_gate_z.bias",
+      "world_model.spatial_gate_r.weight",
+      "world_model.spatial_gate_r.bias",
+      "world_model.spatial_candidate.weight",
+      "world_model.spatial_candidate.bias",
+      "world_model.spatial_summary_proj.0.weight",
+      "world_model.spatial_summary_proj.0.bias",
+      "world_model.spatial_summary_proj.1.weight",
+      "world_model.spatial_summary_proj.1.bias",
+      "world_model.spatial_phase_head.weight",
+      "world_model.spatial_phase_head.bias",
+      "world_model.spatial_support_mode_head.weight",
+      "world_model.spatial_support_mode_head.bias",
+      "world_model.spatial_arm_role_head.weight",
+      "world_model.spatial_arm_role_head.bias",
+      "world_model.spatial_reocclusion_head.weight",
+      "world_model.spatial_reocclusion_head.bias",
+      "world_model.spatial_target_belief_head.weight",
+      "world_model.spatial_target_belief_head.bias",
+      "world_model.spatial_visibility_head.weight",
+      "world_model.spatial_visibility_head.bias",
+      "world_model.spatial_clearance_head.weight",
+      "world_model.spatial_clearance_head.bias",
+      "world_model.spatial_occluder_contact_head.weight",
+      "world_model.spatial_occluder_contact_head.bias",
+      "world_model.spatial_grasp_affordance_head.weight",
+      "world_model.spatial_grasp_affordance_head.bias",
+      "world_model.spatial_support_stability_head.weight",
+      "world_model.spatial_support_stability_head.bias",
+      "world_model.spatial_persistence_head.weight",
+      "world_model.spatial_persistence_head.bias",
+      "world_model.spatial_reocclusion_field_head.weight",
+      "world_model.spatial_reocclusion_field_head.bias",
+      "world_model.spatial_disturbance_head.weight",
+      "world_model.spatial_disturbance_head.bias",
+      "world_model.spatial_uncertainty_head.weight",
+      "world_model.spatial_uncertainty_head.bias",
+      "world_model.spatial_access_head.weight",
+      "world_model.spatial_access_head.bias"
+    ],
+    "unexpected_keys": []
+  }
+}

code/reveal_vla_bimanual/eval/ablations.py

@@ -1,7 +1,9 @@
 MANDATORY_ABLATIONS: tuple[str, ...] = (
-    "no_interaction_head",
-    "no_world_model",
+    "no_geometry",
+    "no_spatial_memory",
+    "compact_world_model",
     "no_planner",
-    "no_role_tokens",
-    "short_history",
+    "gaussian_candidates_only",
+    "no_task_head",
+    "no_support_mode_conditioning",
 )

code/reveal_vla_bimanual/eval/compare_rlbench_sweeps.py

@@ -0,0 +1,143 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+
+def _load_summary(path: Path) -> dict[str, Any]:
+    payload = json.loads(path.read_text(encoding="utf-8"))
+    task_scores = {
+        task_name: float(task_payload.get("mean_success", 0.0))
+        for task_name, task_payload in payload.get("tasks", {}).items()
+    }
+    task_returns = {
+        task_name: float(task_payload.get("mean_return", 0.0))
+        for task_name, task_payload in payload.get("tasks", {}).items()
+    }
+    task_path_recoveries = {
+        task_name: float(sum(task_payload.get("path_recoveries", [])) / max(1, len(task_payload.get("path_recoveries", []))))
+        for task_name, task_payload in payload.get("tasks", {}).items()
+    }
+    task_noop_fallbacks = {
+        task_name: float(sum(task_payload.get("noop_fallbacks", [])) / max(1, len(task_payload.get("noop_fallbacks", []))))
+        for task_name, task_payload in payload.get("tasks", {}).items()
+    }
+    return {
+        "path": str(path),
+        "checkpoint": payload.get("checkpoint"),
+        "mean_success": float(payload.get("mean_success", 0.0)),
+        "mean_return": float(sum(task_returns.values()) / max(1, len(task_returns))),
+        "mean_path_recoveries": float(sum(task_path_recoveries.values()) / max(1, len(task_path_recoveries))),
+        "mean_noop_fallbacks": float(sum(task_noop_fallbacks.values()) / max(1, len(task_noop_fallbacks))),
+        "plan_requested": bool(payload.get("plan_requested", False)),
+        "plan_applied": bool(payload.get("plan_applied", False)),
+        "no_planner": bool(payload.get("no_planner", False)),
+        "no_geometry": bool(payload.get("no_geometry", False)),
+        "disable_task_conditioning": bool(payload.get("disable_task_conditioning", False)),
+        "compact_world_model": bool(payload.get("compact_world_model", False)),
+        "task_scores": task_scores,
+        "task_returns": task_returns,
+        "task_path_recoveries": task_path_recoveries,
+        "task_noop_fallbacks": task_noop_fallbacks,
+        "error_tasks": list(payload.get("error_tasks", [])),
+    }
+
+
+def _pairwise_delta(reference: dict[str, Any], candidate: dict[str, Any]) -> dict[str, Any]:
+    shared_tasks = sorted(set(reference["task_scores"]) & set(candidate["task_scores"]))
+    if not shared_tasks:
+        return {
+            "shared_task_count": 0,
+            "mean_success_delta": 0.0,
+            "mean_return_delta": 0.0,
+            "mean_path_recoveries_delta": 0.0,
+            "mean_noop_fallbacks_delta": 0.0,
+            "per_task_delta": {},
+        }
+    per_task_delta = {
+        task_name: float(candidate["task_scores"][task_name] - reference["task_scores"][task_name])
+        for task_name in shared_tasks
+    }
+    return {
+        "shared_task_count": len(shared_tasks),
+        "mean_success_delta": float(candidate["mean_success"] - reference["mean_success"]),
+        "mean_return_delta": float(candidate["mean_return"] - reference["mean_return"]),
+        "mean_path_recoveries_delta": float(candidate["mean_path_recoveries"] - reference["mean_path_recoveries"]),
+        "mean_noop_fallbacks_delta": float(candidate["mean_noop_fallbacks"] - reference["mean_noop_fallbacks"]),
+        "per_task_delta": per_task_delta,
+    }
+
+
+def _markdown_lines(reference_label: str, comparison: dict[str, Any]) -> list[str]:
+    lines = [
+        "# RLBench Sweep Comparison",
+        "",
+        f"- Reference: `{reference_label}`",
+        "",
+        "## Runs",
+        "",
+    ]
+    for label, payload in comparison["runs"].items():
+        lines.append(
+            f"- `{label}`: mean_success={payload['mean_success']:.3f}, "
+            f"mean_return={payload['mean_return']:.3f}, "
+            f"mean_path_recoveries={payload['mean_path_recoveries']:.3f}, "
+            f"mean_noop_fallbacks={payload['mean_noop_fallbacks']:.3f}, "
+            f"plan_applied={payload['plan_applied']}, "
+            f"errors={len(payload['error_tasks'])}, "
+            f"path=`{payload['path']}`"
+        )
+    lines.extend(["", "## Pairwise Deltas", ""])
+    for label, payload in comparison["pairwise_against_reference"].items():
+        lines.append(
+            f"- `{label}`: mean_success_delta={payload['mean_success_delta']:.3f}, "
+            f"mean_return_delta={payload['mean_return_delta']:.3f}, "
+            f"mean_path_recoveries_delta={payload['mean_path_recoveries_delta']:.3f}, "
+            f"mean_noop_fallbacks_delta={payload['mean_noop_fallbacks_delta']:.3f}, "
+            f"shared_tasks={payload['shared_task_count']}"
+        )
+    return lines
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run", action="append", required=True, help="label=/abs/path/to/rollout_eval.json")
+    parser.add_argument("--reference-label", required=True)
+    parser.add_argument("--output-dir", required=True)
+    args = parser.parse_args()
+
+    runs: dict[str, dict[str, Any]] = {}
+    for item in args.run:
+        label, raw_path = item.split("=", 1)
+        runs[label] = _load_summary(Path(raw_path).resolve())
+
+    if args.reference_label not in runs:
+        raise ValueError(f"Missing reference label {args.reference_label!r} in provided runs.")
+
+    reference = runs[args.reference_label]
+    comparison = {
+        "reference_label": args.reference_label,
+        "runs": runs,
+        "pairwise_against_reference": {
+            label: _pairwise_delta(reference, payload)
+            for label, payload in runs.items()
+            if label != args.reference_label
+        },
+    }
+
+    output_dir = Path(args.output_dir).resolve()
+    output_dir.mkdir(parents=True, exist_ok=True)
+    (output_dir / "rlbench_comparison.json").write_text(
+        json.dumps(comparison, indent=2),
+        encoding="utf-8",
+    )
+    (output_dir / "rlbench_comparison.md").write_text(
+        "\n".join(_markdown_lines(args.reference_label, comparison)) + "\n",
+        encoding="utf-8",
+    )
+
+
+if __name__ == "__main__":
+    main()

code/reveal_vla_bimanual/eval/run_ablations.py

@@ -7,7 +7,7 @@ import time
 
 from eval.ablations import MANDATORY_ABLATIONS
 from eval.report import write_comparison_report
-from eval.run_reveal_benchmark import evaluate_model, load_model
+from eval.run_reveal_benchmark import _paired_seed_summary, evaluate_model, load_model
 from sim_reveal import available_proxy_names
 
 import torch
@@ -20,6 +20,7 @@ def main() -> None:
     parser.add_argument("--resolution", type=int, default=None)
     parser.add_argument("--output-root", default="/workspace/reports/reveal_ablation")
     parser.add_argument("--proxies", nargs="*", default=None)
+    parser.add_argument("--chunk-commit-steps", type=int, default=0)
     parser.add_argument("--resume", action="store_true")
     args = parser.parse_args()
 
@@ -29,17 +30,20 @@ def main() -> None:
     proxies = list(args.proxies or available_proxy_names())
     output_root = Path(args.output_root)
     output_root.mkdir(parents=True, exist_ok=True)
+    chunk_commit_steps = None if args.chunk_commit_steps <= 0 else args.chunk_commit_steps
 
     json_path = output_root / "ablations.json"
     partial_path = output_root / "ablations.partial.json"
     sections = {}
     raw = {}
+    full_episode_records: list[dict[str, float | int | str]] | None = None
     completed_labels: set[str] = set()
     if args.resume and partial_path.exists():
         partial = json.loads(partial_path.read_text(encoding="utf-8"))
         raw = partial.get("raw", {})
         sections = partial.get("sections", {})
         completed_labels = set(raw)
+        full_episode_records = raw.get("full_model", {}).get("episode_records")
         print(json.dumps({"resume_from": str(partial_path), "completed": sorted(completed_labels)}, indent=2))
 
     ablations = (None, *MANDATORY_ABLATIONS)
@@ -56,7 +60,9 @@ def main() -> None:
             episodes=args.episodes,
             resolution=resolution,
             ablation=ablation,
+            chunk_commit_steps=chunk_commit_steps,
         )
+        metrics, episode_records = metrics
         raw[label] = {
             "per_task_success": metrics.per_task_success,
             "mean_success": metrics.mean_success,
@@ -65,6 +71,7 @@ def main() -> None:
             "reocclusion_rate": metrics.reocclusion_rate,
             "persistence_horizon_mae": metrics.persistence_horizon_mae,
             "disturbance_cost": metrics.disturbance_cost,
+            "episode_records": episode_records,
         }
         sections[label] = {
             "mean_success": metrics.mean_success,
@@ -73,12 +80,21 @@ def main() -> None:
             "reocclusion_rate": metrics.reocclusion_rate or 0.0,
             "persistence_horizon_mae": metrics.persistence_horizon_mae or 0.0,
             "disturbance_cost": metrics.disturbance_cost or 0.0,
+            "chunk_commit_steps": float(0 if chunk_commit_steps is None else chunk_commit_steps),
         }
+        if label == "full_model":
+            full_episode_records = episode_records
+        elif full_episode_records is not None:
+            paired = _paired_seed_summary(full_episode_records, episode_records)
+            raw[label]["paired_seed_summary_vs_full_model"] = paired
+            for key, value in paired.items():
+                sections[label][f"paired_{key}_vs_full_model"] = value
         partial_path.write_text(
             json.dumps(
                 {
                     "checkpoint": args.checkpoint,
                     "episodes": args.episodes,
+                    "chunk_commit_steps": 0 if chunk_commit_steps is None else chunk_commit_steps,
                     "sections": sections,
                     "raw": raw,
                     "elapsed_seconds": time.monotonic() - start_time,

code/reveal_vla_bimanual/eval/run_peract2_task_sweep.py

@@ -25,6 +25,10 @@ def _run_task(
     chunk_commit_steps: int,
     allow_unsupervised_planning: bool,
     disable_support_mode_conditioning: bool,
+    disable_task_conditioning: bool,
+    no_geometry: bool,
+    compact_world_model: bool,
+    no_planner: bool,
 ) -> dict[str, Any]:
     task_dir = output_dir / task_name
     task_dir.mkdir(parents=True, exist_ok=True)
@@ -53,10 +57,18 @@ def _run_task(
         command.append("--headless")
     if plan:
         command.append("--plan")
+    if no_planner:
+        command.append("--no-planner")
     if allow_unsupervised_planning:
         command.append("--allow-unsupervised-planning")
     if disable_support_mode_conditioning:
         command.append("--disable-support-mode-conditioning")
+    if disable_task_conditioning:
+        command.append("--disable-task-conditioning")
+    if no_geometry:
+        command.append("--no-geometry")
+    if compact_world_model:
+        command.append("--compact-world-model")
 
     completed = subprocess.run(
         command,
@@ -105,6 +117,10 @@ def _write_summary_markdown(path: Path, payload: dict[str, Any]) -> None:
         f"- Episodes per task: `{payload['episodes_per_task']}`",
         f"- Episode length: `{payload['episode_length']}`",
         f"- Resolution: `{payload['resolution']}`",
+        f"- No planner: `{payload['no_planner']}`",
+        f"- Disable task conditioning: `{payload['disable_task_conditioning']}`",
+        f"- No geometry: `{payload['no_geometry']}`",
+        f"- Compact world model: `{payload['compact_world_model']}`",
         f"- Task count: `{payload['task_count']}`",
         f"- Error tasks: `{payload['error_tasks']}`",
         f"- Mean success: `{payload['mean_success']:.3f}`",
@@ -144,6 +160,10 @@ def _run_mode(args: argparse.Namespace, plan: bool) -> Path:
         "episode_length": args.episode_length,
         "resolution": args.resolution,
         "device": args.device,
+        "no_planner": args.no_planner,
+        "disable_task_conditioning": args.disable_task_conditioning,
+        "no_geometry": args.no_geometry,
+        "compact_world_model": args.compact_world_model,
         "tasks": {},
         "subprocess_mode": "isolated_per_task",
     }
@@ -165,6 +185,10 @@ def _run_mode(args: argparse.Namespace, plan: bool) -> Path:
             chunk_commit_steps=args.chunk_commit_steps,
             allow_unsupervised_planning=args.allow_unsupervised_planning,
             disable_support_mode_conditioning=args.disable_support_mode_conditioning,
+            disable_task_conditioning=args.disable_task_conditioning,
+            no_geometry=args.no_geometry,
+            compact_world_model=args.compact_world_model,
+            no_planner=args.no_planner,
         )
 
     task_scores = [float(task_payload["mean_success"]) for task_payload in summary["tasks"].values()]
@@ -194,6 +218,10 @@ def main() -> None:
     parser.add_argument("--chunk-commit-steps", type=int, default=4)
     parser.add_argument("--allow-unsupervised-planning", action="store_true")
     parser.add_argument("--disable-support-mode-conditioning", action="store_true")
+    parser.add_argument("--disable-task-conditioning", action="store_true")
+    parser.add_argument("--no-geometry", action="store_true")
+    parser.add_argument("--compact-world-model", action="store_true")
+    parser.add_argument("--no-planner", action="store_true")
     parser.add_argument("--skip-noplan", action="store_true")
     parser.add_argument("--skip-plan", action="store_true")
     args = parser.parse_args()

code/reveal_vla_bimanual/eval/run_reveal_benchmark.py

@@ -57,7 +57,12 @@ def load_model(checkpoint_path: str | Path, device: torch.device) -> tuple[torch
     allowed_missing = {
         key
         for key in incompatible.missing_keys
-        if key.startswith("memory.action_proj.") or key.endswith("arm_identity.weight")
+        if key.startswith("memory.action_proj.")
+        or key.endswith("arm_identity.weight")
+        or key.endswith("task_embedding.weight")
+        or key.startswith("elastic_state_head.decoder.task_")
+        or key.startswith("world_model.task_")
+        or key.startswith("world_model.spatial_")
     }
     missing_other = sorted(set(incompatible.missing_keys) - allowed_missing)
     if missing_other or incompatible.unexpected_keys:
@@ -175,11 +180,18 @@ def select_chunk(
             history_depths=batch.get("history_depths"),
             history_depth_valid=batch.get("history_depth_valid"),
             plan=True,
+            support_mode_conditioning=(ablation != "no_support_mode_conditioning"),
             use_world_model=(ablation not in {"no_world_model", "no_planner"}),
             use_planner=(ablation != "no_planner"),
             use_depth=(ablation != "no_depth"),
+            use_geometry_tokens=(ablation != "no_geometry"),
+            use_camera_pose_tokens=(ablation != "no_camera_pose"),
             use_role_tokens=(ablation not in {"no_role_tokens", "no_role_symmetry"}),
             history_steps_override=(2 if ablation == "short_history" else None),
+            use_memory=(ablation != "no_spatial_memory"),
+            use_task_conditioning=(ablation != "no_task_head"),
+            rollout_mode_override=("compact_rollout" if ablation == "compact_world_model" else None),
+            use_proposal_candidates=(ablation != "gaussian_candidates_only"),
         )
         if "planned_chunk" in outputs and ablation != "no_planner":
             return outputs["planned_chunk"], outputs
@@ -204,6 +216,52 @@ def select_chunk(
     return outputs["action_mean"], outputs
 
 
+def _bootstrap_interval(values: list[float], bootstrap_samples: int = 1000) -> dict[str, float]:
+    if not values:
+        return {"mean": 0.0, "low": 0.0, "high": 0.0}
+    array = np.asarray(values, dtype=np.float32)
+    mean = float(array.mean())
+    if array.size == 1:
+        return {"mean": mean, "low": mean, "high": mean}
+    rng = np.random.default_rng(0)
+    sample_indices = rng.integers(0, array.size, size=(bootstrap_samples, array.size))
+    sampled_means = array[sample_indices].mean(axis=1)
+    return {
+        "mean": mean,
+        "low": float(np.percentile(sampled_means, 2.5)),
+        "high": float(np.percentile(sampled_means, 97.5)),
+    }
+
+
+def _paired_seed_summary(
+    reference_records: list[dict[str, float | int | str]],
+    candidate_records: list[dict[str, float | int | str]],
+) -> dict[str, float]:
+    reference_by_key = {
+        (str(record["proxy_name"]), int(record["seed"])): record for record in reference_records
+    }
+    success_deltas = []
+    visibility_deltas = []
+    reocclusion_deltas = []
+    disturbance_deltas = []
+    for record in candidate_records:
+        key = (str(record["proxy_name"]), int(record["seed"]))
+        baseline = reference_by_key.get(key)
+        if baseline is None:
+            continue
+        success_deltas.append(float(record["success"]) - float(baseline["success"]))
+        visibility_deltas.append(float(record["visibility_integral"]) - float(baseline["visibility_integral"]))
+        reocclusion_deltas.append(float(record["reocclusion_rate"]) - float(baseline["reocclusion_rate"]))
+        disturbance_deltas.append(float(record["disturbance_cost"]) - float(baseline["disturbance_cost"]))
+    return {
+        "paired_episodes": float(len(success_deltas)),
+        "success_delta": float(np.mean(success_deltas)) if success_deltas else 0.0,
+        "visibility_delta": float(np.mean(visibility_deltas)) if visibility_deltas else 0.0,
+        "reocclusion_delta": float(np.mean(reocclusion_deltas)) if reocclusion_deltas else 0.0,
+        "disturbance_delta": float(np.mean(disturbance_deltas)) if disturbance_deltas else 0.0,
+    }
+
+
 def evaluate_model(
     model: torch.nn.Module,
     device: torch.device,
@@ -212,24 +270,26 @@ def evaluate_model(
     resolution: int,
     ablation: str | None = None,
     chunk_commit_steps: int | None = None,
-) -> BenchmarkMetrics:
+) -> tuple[BenchmarkMetrics, list[dict[str, float | int | str]]]:
     per_task_success: dict[str, float] = {}
     visibility_scores = []
     corridor_scores = []
     reocclusion_scores = []
     persistence_errors = []
     disturbance_scores = []
+    episode_records: list[dict[str, float | int | str]] = []
     history_steps = int(getattr(model.config.memory, "history_steps", 0)) if hasattr(model, "config") else 0
 
     for proxy_offset, proxy_name in enumerate(proxies):
         successes = []
         for episode_idx in range(episodes):
+            seed = proxy_offset * 10_000 + episode_idx
             env = make_proxy_env(
                 proxy_name=proxy_name,
                 resolution=resolution,
-                seed=proxy_offset * 10_000 + episode_idx,
+                seed=seed,
             )
-            observation, privileged_state = env.reset(seed=proxy_offset * 10_000 + episode_idx)
+            observation, privileged_state = env.reset(seed=seed)
             episode_visibility = [float(privileged_state["visibility"])]
             episode_corridor = [float(privileged_state["corridor_feasible"][privileged_state["support_mode"]].any())]
             episode_disturbance = [float(privileged_state["disturbance_cost"])]
@@ -287,20 +347,39 @@ def evaluate_model(
                     if done:
                         break
             successes.append(float(privileged_state["retrieval_success"]))
-            visibility_scores.append(visibility_integral(np.asarray(episode_visibility)))
-            corridor_scores.append(corridor_availability(np.asarray(episode_corridor)))
-            reocclusion_scores.append(reocclusion_rate(np.asarray(episode_corridor)))
-            disturbance_scores.append(mean_disturbance_cost(np.asarray(episode_disturbance)))
+            episode_visibility_integral = visibility_integral(np.asarray(episode_visibility))
+            episode_corridor_availability = corridor_availability(np.asarray(episode_corridor))
+            episode_reocclusion = reocclusion_rate(np.asarray(episode_corridor))
+            episode_disturbance_cost = mean_disturbance_cost(np.asarray(episode_disturbance))
+            visibility_scores.append(episode_visibility_integral)
+            corridor_scores.append(episode_corridor_availability)
+            reocclusion_scores.append(episode_reocclusion)
+            disturbance_scores.append(episode_disturbance_cost)
+            episode_records.append(
+                {
+                    "proxy_name": proxy_name,
+                    "seed": seed,
+                    "episode_index": episode_idx,
+                    "success": float(privileged_state["retrieval_success"]),
+                    "visibility_integral": episode_visibility_integral,
+                    "corridor_availability": episode_corridor_availability,
+                    "reocclusion_rate": episode_reocclusion,
+                    "disturbance_cost": episode_disturbance_cost,
+                }
+            )
         per_task_success[proxy_name] = float(np.mean(successes))
 
-    return BenchmarkMetrics(
-        per_task_success=per_task_success,
-        mean_success=mean_success(per_task_success),
-        visibility_integral=float(np.mean(visibility_scores)) if visibility_scores else None,
-        corridor_availability=float(np.mean(corridor_scores)) if corridor_scores else None,
-        reocclusion_rate=float(np.mean(reocclusion_scores)) if reocclusion_scores else None,
-        persistence_horizon_mae=float(np.mean(persistence_errors)) if persistence_errors else None,
-        disturbance_cost=float(np.mean(disturbance_scores)) if disturbance_scores else None,
+    return (
+        BenchmarkMetrics(
+            per_task_success=per_task_success,
+            mean_success=mean_success(per_task_success),
+            visibility_integral=float(np.mean(visibility_scores)) if visibility_scores else None,
+            corridor_availability=float(np.mean(corridor_scores)) if corridor_scores else None,
+            reocclusion_rate=float(np.mean(reocclusion_scores)) if reocclusion_scores else None,
+            persistence_horizon_mae=float(np.mean(persistence_errors)) if persistence_errors else None,
+            disturbance_cost=float(np.mean(disturbance_scores)) if disturbance_scores else None,
+        ),
+        episode_records,
     )
 
 
@@ -319,7 +398,8 @@ def _metrics_to_dict(metrics: BenchmarkMetrics) -> dict[str, float | dict[str, f
 def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--model", action="append", required=True, help="label=/abs/path/checkpoint.pt")
-    parser.add_argument("--episodes", type=int, default=24)
+    parser.add_argument("--episodes", type=int, default=None)
+    parser.add_argument("--benchmark-mode", choices=("smoke", "serious"), default="smoke")
     parser.add_argument("--resolution", type=int, default=None)
     parser.add_argument("--ablation", default=None)
     parser.add_argument("--output-root", default="/workspace/reports/reveal_eval")
@@ -329,37 +409,76 @@ def main() -> None:
 
     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
     proxies = list(args.proxies or available_proxy_names())
+    episodes = int(args.episodes or (100 if args.benchmark_mode == "serious" else 24))
+    chunk_commit_steps = None if args.chunk_commit_steps <= 0 else args.chunk_commit_steps
     output_root = Path(args.output_root)
     output_root.mkdir(parents=True, exist_ok=True)
 
     sections: dict[str, dict[str, float | str]] = {}
-    raw_metrics: dict[str, dict[str, float | dict[str, float]]] = {}
+    raw_metrics: dict[str, dict[str, Any]] = {
+        "benchmark_config": {
+            "episodes": episodes,
+            "benchmark_mode": args.benchmark_mode,
+            "ablation": args.ablation,
+            "proxies": proxies,
+            "chunk_commit_steps": 0 if chunk_commit_steps is None else chunk_commit_steps,
+        }
+    }
+    episode_records_by_label: dict[str, list[dict[str, float | int | str]]] = {}
     for item in args.model:
         label, checkpoint_path = item.split("=", maxsplit=1)
         model, checkpoint = load_model(checkpoint_path, device=device)
         resolution = int(args.resolution or checkpoint.get("data_resolution", 96))
-        metrics = evaluate_model(
+        metrics, episode_records = evaluate_model(
             model=model,
             device=device,
             proxies=proxies,
-            episodes=args.episodes,
+            episodes=episodes,
             resolution=resolution,
             ablation=args.ablation,
-            chunk_commit_steps=(None if args.chunk_commit_steps <= 0 else args.chunk_commit_steps),
+            chunk_commit_steps=chunk_commit_steps,
         )
-        raw_metrics[label] = _metrics_to_dict(metrics)
+        episode_records_by_label[label] = episode_records
+        success_values = [float(record["success"]) for record in episode_records]
+        visibility_values = [float(record["visibility_integral"]) for record in episode_records]
+        reocclusion_values = [float(record["reocclusion_rate"]) for record in episode_records]
+        disturbance_values = [float(record["disturbance_cost"]) for record in episode_records]
+        raw_metrics[label] = {
+            **_metrics_to_dict(metrics),
+            "bootstrap_ci": {
+                "success": _bootstrap_interval(success_values),
+                "visibility_integral": _bootstrap_interval(visibility_values),
+                "reocclusion_rate": _bootstrap_interval(reocclusion_values),
+                "disturbance_cost": _bootstrap_interval(disturbance_values),
+            },
+            "episode_records": episode_records,
+        }
         sections[label] = {
             "checkpoint": checkpoint_path,
+            "benchmark_mode": args.benchmark_mode,
+            "episodes": float(episodes),
+            "chunk_commit_steps": float(0 if chunk_commit_steps is None else chunk_commit_steps),
             "mean_success": metrics.mean_success,
             "visibility_integral": metrics.visibility_integral or 0.0,
             "corridor_availability": metrics.corridor_availability or 0.0,
             "reocclusion_rate": metrics.reocclusion_rate or 0.0,
             "persistence_horizon_mae": metrics.persistence_horizon_mae or 0.0,
             "disturbance_cost": metrics.disturbance_cost or 0.0,
+            "success_ci_low": raw_metrics[label]["bootstrap_ci"]["success"]["low"],
+            "success_ci_high": raw_metrics[label]["bootstrap_ci"]["success"]["high"],
         }
         for task_name, score in metrics.per_task_success.items():
             sections[label][f"{task_name}_success"] = score
 
+    labels = [item.split("=", maxsplit=1)[0] for item in args.model]
+    if labels:
+        reference_label = labels[0]
+        for label in labels[1:]:
+            summary = _paired_seed_summary(episode_records_by_label[reference_label], episode_records_by_label[label])
+            raw_metrics[label]["paired_seed_summary_vs_" + reference_label] = summary
+            for key, value in summary.items():
+                sections[label][f"paired_{key}_vs_{reference_label}"] = value
+
     json_path = output_root / "reveal_benchmark.json"
     json_path.write_text(json.dumps(raw_metrics, indent=2), encoding="utf-8")
     write_comparison_report(output_root / "reveal_benchmark.md", "Reveal Proxy Benchmark", sections)

code/reveal_vla_bimanual/eval/run_rlbench_rollout_eval.py

@@ -122,11 +122,15 @@ def main() -> None:
     parser.add_argument("--resolution", type=int, default=224)
     parser.add_argument("--device", default="cuda")
     parser.add_argument("--plan", action="store_true")
+    parser.add_argument("--no-planner", action="store_true")
     parser.add_argument("--allow-unsupervised-planning", action="store_true")
     parser.add_argument("--disable-support-mode-conditioning", action="store_true")
+    parser.add_argument("--disable-task-conditioning", action="store_true")
     parser.add_argument("--headless", action="store_true", default=True)
     parser.add_argument("--chunk-commit-steps", type=int, default=0)
     parser.add_argument("--reset-retries", type=int, default=20)
+    parser.add_argument("--no-geometry", action="store_true")
+    parser.add_argument("--compact-world-model", action="store_true")
     args = parser.parse_args()
 
     checkpoint = torch.load(Path(args.checkpoint), map_location="cpu", weights_only=False)
@@ -138,7 +142,12 @@ def main() -> None:
     allowed_missing = {
         key
         for key in incompatible.missing_keys
-        if key.startswith("memory.action_proj.") or key.endswith("arm_identity.weight")
+        if key.startswith("memory.action_proj.")
+        or key.endswith("arm_identity.weight")
+        or key.endswith("task_embedding.weight")
+        or key.startswith("elastic_state_head.decoder.task_")
+        or key.startswith("world_model.task_")
+        or key.startswith("world_model.spatial_")
     }
     missing_other = sorted(set(incompatible.missing_keys) - allowed_missing)
     if missing_other or incompatible.unexpected_keys:
@@ -147,7 +156,7 @@ def main() -> None:
             f"Missing keys: {missing_other}. Unexpected keys: {list(incompatible.unexpected_keys)}"
         )
     model.eval()
-    plan_requested = bool(args.plan)
+    plan_requested = bool(args.plan) and not bool(args.no_planner)
     plan_applied = plan_requested and planner_enabled(trainer_config, during_eval=True)
     planning_note = None
     if plan_requested and not policy_supports_planning(trainer_config.policy_type):
@@ -166,6 +175,9 @@ def main() -> None:
         "plan_applied": plan_applied,
         "planner_mode": trainer_config.planner_mode,
         "support_mode_conditioning": not args.disable_support_mode_conditioning,
+        "task_conditioning": not args.disable_task_conditioning,
+        "geometry_enabled": not args.no_geometry,
+        "world_model_mode": "compact_rollout" if args.compact_world_model else "checkpoint_default",
         "episodes_per_task": args.episodes_per_task,
         "episode_length": args.episode_length,
         "resolution": args.resolution,
@@ -196,6 +208,9 @@ def main() -> None:
             env.launch()
             task = env.get_task(task_class)
             task_reset_retries: list[int] = []
+            task_path_recoveries: list[int] = []
+            task_noop_fallbacks: list[int] = []
+            task_episode_traces: list[dict[str, Any]] = []
             for _ in range(args.episodes_per_task):
                 descriptions, obs, reset_retries = _reset_task_with_retries(task, max_attempts=max(1, args.reset_retries))
                 task_reset_retries.append(int(reset_retries))
@@ -204,6 +219,7 @@ def main() -> None:
                 success = 0.0
                 episode_recoveries = 0
                 episode_noop_fallbacks = 0
+                episode_trace: dict[str, Any] = {"language_goal": language_goal, "steps": []}
                 history_images: list[np.ndarray] = []
                 history_proprio: list[np.ndarray] = []
                 history_actions: list[np.ndarray] = []
@@ -256,6 +272,10 @@ def main() -> None:
                                 history_actions=history_actions_tensor,
                                 plan=plan_applied,
                                 support_mode_conditioning=not args.disable_support_mode_conditioning,
+                                use_planner=not args.no_planner,
+                                use_geometry_tokens=not args.no_geometry,
+                                use_task_conditioning=not args.disable_task_conditioning,
+                                rollout_mode_override=("compact_rollout" if args.compact_world_model else None),
                             )
                         else:
                             outputs = model(
@@ -269,6 +289,20 @@ def main() -> None:
                     chosen_chunk = outputs["action_mean"]
                     if plan_applied and "planned_chunk" in outputs:
                         chosen_chunk = outputs["planned_chunk"]
+                    best_local = 0
+                    if isinstance(outputs.get("ranking_diagnostics"), dict) and "best_local_indices" in outputs["ranking_diagnostics"]:
+                        best_local = int(outputs["ranking_diagnostics"]["best_local_indices"][0].detach().cpu().item())
+                    chosen_macro_mode = None
+                    if "planner_topk_mode_names" in outputs and outputs["planner_topk_mode_names"]:
+                        chosen_macro_mode = outputs["planner_topk_mode_names"][0][best_local]
+                    predicted_reocclusion = None
+                    if "planned_rollout" in outputs and "reocclusion_field" in outputs["planned_rollout"]:
+                        predicted_reocclusion = float(
+                            outputs["planned_rollout"]["reocclusion_field"][0, best_local].mean().detach().cpu().item()
+                        )
+                    planner_scores = None
+                    if "planner_scores" in outputs:
+                        planner_scores = outputs["planner_scores"][0].detach().cpu().tolist()
                     chunk_np = chosen_chunk[0].detach().float().cpu().numpy()
                     commit_steps = chunk_np.shape[0] if args.chunk_commit_steps <= 0 else min(args.chunk_commit_steps, chunk_np.shape[0])
                     done = False
@@ -292,6 +326,17 @@ def main() -> None:
                         obs, reward, done, recovered_steps, noop_fallbacks = _step_bimanual_chunk(task, obs, step_action)
                         episode_recoveries += recovered_steps
                         episode_noop_fallbacks += noop_fallbacks
+                        episode_trace["steps"].append(
+                            {
+                                "timestep": int(timestep),
+                                "chosen_macro_mode": chosen_macro_mode,
+                                "planner_scores": planner_scores,
+                                "predicted_reocclusion": predicted_reocclusion,
+                                "support_mode_conditioning": not args.disable_support_mode_conditioning,
+                                "path_recoveries": int(recovered_steps),
+                                "noop_fallbacks": int(noop_fallbacks),
+                            }
+                        )
                         total_reward += float(reward)
                         timestep += 1
                         if reward >= 1.0:
@@ -302,13 +347,21 @@ def main() -> None:
                         break
                 task_successes.append(success)
                 task_returns.append(total_reward)
+                task_path_recoveries.append(int(episode_recoveries))
+                task_noop_fallbacks.append(int(episode_noop_fallbacks))
+                episode_trace["success"] = float(success)
+                episode_trace["return"] = float(total_reward)
+                episode_trace["path_recoveries"] = int(episode_recoveries)
+                episode_trace["noop_fallbacks"] = int(episode_noop_fallbacks)
+                task_episode_traces.append(episode_trace)
             results["tasks"][task_name] = {
                 "task_class": task_class.__name__,
                 "successes": task_successes,
                 "returns": task_returns,
-                "path_recoveries": episode_recoveries if args.episodes_per_task == 1 else None,
-                "noop_fallbacks": episode_noop_fallbacks if args.episodes_per_task == 1 else None,
+                "path_recoveries": task_path_recoveries,
+                "noop_fallbacks": task_noop_fallbacks,
                 "reset_retries": task_reset_retries,
+                "episode_traces": task_episode_traces,
                 "mean_success": float(np.mean(task_successes)) if task_successes else 0.0,
                 "mean_return": float(np.mean(task_returns)) if task_returns else 0.0,
             }
@@ -331,6 +384,9 @@ def main() -> None:
         f"- Plan requested: `{results['plan_requested']}`",
         f"- Plan applied: `{results['plan_applied']}`",
         f"- Support-mode conditioning: `{results['support_mode_conditioning']}`",
+        f"- Task conditioning: `{results['task_conditioning']}`",
+        f"- Geometry enabled: `{results['geometry_enabled']}`",
+        f"- World-model mode: `{results['world_model_mode']}`",
         f"- Mean success: `{results['mean_success']:.3f}`",
         "",
         "## Per-task",

code/reveal_vla_bimanual/eval/run_teacher_audit.py

@@ -0,0 +1,115 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+
+from eval.report import write_comparison_report
+from sim_reveal.procedural_envs import available_proxy_names, make_proxy_env
+
+
+BASELINES: tuple[str, ...] = ("teacher", "reveal_only", "retrieve_only", "no_hold", "random")
+
+
+def _evaluate_baseline(
+    proxy_name: str,
+    baseline_name: str,
+    episodes: int,
+    resolution: int,
+    chunk_horizon: int,
+    rollout_horizon: int,
+) -> dict[str, float]:
+    success = []
+    reveal = []
+    visibility = []
+    hold = []
+    reocclusion = []
+    disturbance = []
+    utility = []
+    for seed in range(episodes):
+        env = make_proxy_env(proxy_name=proxy_name, resolution=resolution, seed=seed, rollout_horizon=rollout_horizon)
+        _, _ = env.reset(seed=seed)
+        chunk = env.baseline_action_chunk(baseline_name, chunk_horizon=chunk_horizon)
+        outcome = env.evaluate_action_chunk(chunk, rollout_horizon=rollout_horizon)
+        success_value = float(outcome["retrieval_success"])
+        reveal_value = float(outcome["reveal_achieved"])
+        visibility_value = float(outcome["visibility_integral"])
+        hold_value = float(outcome["hold_persistence"])
+        reocclusion_value = float(outcome["reocclusion_rate"])
+        disturbance_value = float(outcome["final_disturbance_cost"])
+        utility_value = success_value + 0.25 * reveal_value + 0.1 * hold_value + 0.05 * visibility_value
+        utility_value -= reocclusion_value + disturbance_value
+        success.append(success_value)
+        reveal.append(reveal_value)
+        visibility.append(visibility_value)
+        hold.append(hold_value)
+        reocclusion.append(reocclusion_value)
+        disturbance.append(disturbance_value)
+        utility.append(utility_value)
+    return {
+        "success": float(np.mean(success)),
+        "reveal_achieved": float(np.mean(reveal)),
+        "visibility_integral": float(np.mean(visibility)),
+        "hold_persistence": float(np.mean(hold)),
+        "reocclusion_rate": float(np.mean(reocclusion)),
+        "disturbance_cost": float(np.mean(disturbance)),
+        "utility": float(np.mean(utility)),
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--episodes", type=int, default=100)
+    parser.add_argument("--resolution", type=int, default=96)
+    parser.add_argument("--chunk-horizon", type=int, default=8)
+    parser.add_argument("--rollout-horizon", type=int, default=5)
+    parser.add_argument("--output-root", default="/workspace/reports/reveal_teacher_audit")
+    parser.add_argument("--proxies", nargs="*", default=None)
+    args = parser.parse_args()
+
+    proxies = list(args.proxies or available_proxy_names())
+    output_root = Path(args.output_root)
+    output_root.mkdir(parents=True, exist_ok=True)
+
+    raw: dict[str, dict[str, dict[str, float]]] = {}
+    sections: dict[str, dict[str, float | str]] = {}
+    for proxy_name in proxies:
+        proxy_results = {
+            baseline_name: _evaluate_baseline(
+                proxy_name=proxy_name,
+                baseline_name=baseline_name,
+                episodes=args.episodes,
+                resolution=args.resolution,
+                chunk_horizon=args.chunk_horizon,
+                rollout_horizon=args.rollout_horizon,
+            )
+            for baseline_name in BASELINES
+        }
+        raw[proxy_name] = proxy_results
+        teacher_metrics = proxy_results["teacher"]
+        for baseline_name in BASELINES[1:]:
+            baseline_metrics = proxy_results[baseline_name]
+            label = f"{proxy_name}:{baseline_name}"
+            sections[label] = {
+                "episodes": float(args.episodes),
+                "teacher_success": teacher_metrics["success"],
+                "baseline_success": baseline_metrics["success"],
+                "teacher_utility": teacher_metrics["utility"],
+                "baseline_utility": baseline_metrics["utility"],
+                "success_delta": teacher_metrics["success"] - baseline_metrics["success"],
+                "utility_delta": teacher_metrics["utility"] - baseline_metrics["utility"],
+                "hold_delta": teacher_metrics["hold_persistence"] - baseline_metrics["hold_persistence"],
+                "reocclusion_delta": teacher_metrics["reocclusion_rate"] - baseline_metrics["reocclusion_rate"],
+                "disturbance_delta": teacher_metrics["disturbance_cost"] - baseline_metrics["disturbance_cost"],
+            }
+
+    json_path = output_root / "teacher_audit.json"
+    json_path.write_text(json.dumps(raw, indent=2), encoding="utf-8")
+    write_comparison_report(output_root / "teacher_audit.md", "Reveal Teacher Audit", sections)
+    print(json.dumps({"output_json": str(json_path), "sections": sections}, indent=2))
+
+
+if __name__ == "__main__":
+    main()

code/reveal_vla_bimanual/models/action_decoder.py

@@ -394,6 +394,79 @@ DEFAULT_PROPOSAL_MODES = (
     "retrieve",
 )
 
+TASK_PROPOSAL_MODES = {
+    "foliage": (
+        "sweep_left",
+        "sweep_right",
+        "pin_canopy",
+        "widen_gap",
+        "maintain_gap",
+        "insert_actor",
+        "retrieve",
+    ),
+    "bag": (
+        "pin_left_rim",
+        "pin_right_rim",
+        "widen_mouth",
+        "maintain_mouth",
+        "probe_inside",
+        "insert_actor",
+        "retrieve",
+    ),
+    "cloth": (
+        "lift_edge",
+        "separate_layer",
+        "stabilize_fold",
+        "maintain_lift",
+        "insert_actor",
+        "retrieve",
+    ),
+}
+
+TASK_INDEX = {"foliage": 0, "bag": 1, "cloth": 2}
+
+
+def infer_task_name_from_text(text: str | None) -> str:
+    if not text:
+        return "generic"
+    lowered = text.lower()
+    if any(token in lowered for token in ("foliage", "canopy", "leaf", "leaves", "snail")):
+        return "foliage"
+    if any(token in lowered for token in ("bag", "mouth", "rim", "aperture")):
+        return "bag"
+    if any(token in lowered for token in ("cloth", "fold", "layer", "suitcase", "garment")):
+        return "cloth"
+    return "generic"
+
+
+def proposal_mode_vocab(task_name: str, num_modes: int) -> tuple[str, ...]:
+    if task_name == "generic":
+        base_vocab = tuple(DEFAULT_PROPOSAL_MODES)
+    else:
+        vocab = TASK_PROPOSAL_MODES[task_name]
+        if len(vocab) > num_modes:
+            if num_modes >= 6:
+                base_vocab = (
+                    vocab[0],
+                    vocab[1],
+                    vocab[2],
+                    vocab[3],
+                    vocab[-2],
+                    vocab[-1],
+                )[:num_modes]
+            else:
+                base_vocab = vocab[:num_modes]
+        else:
+            base_vocab = vocab
+    if len(base_vocab) >= num_modes:
+        return tuple(base_vocab[:num_modes])
+    if not base_vocab:
+        return tuple("retrieve" for _ in range(num_modes))
+    padded = list(base_vocab)
+    while len(padded) < num_modes:
+        padded.append(base_vocab[-1])
+    return tuple(padded)
+
 
 def swap_arm_action_order(action_chunk: Tensor) -> Tensor:
     midpoint = action_chunk.shape[-1] // 2
@@ -416,6 +489,7 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         self.arm_decoder = nn.TransformerDecoder(decoder_layer, num_layers=config.num_layers)
         self.query_embed = nn.Embedding(config.chunk_size, config.hidden_dim)
         self.arm_identity = nn.Embedding(2, config.hidden_dim)
+        self.task_embedding = nn.Embedding(len(TASK_INDEX), config.hidden_dim)
         self.phase_adapter = nn.Linear(config.num_phases, config.hidden_dim)
         self.role_adapter = nn.Linear(config.num_arm_roles, config.hidden_dim)
         self.context_proj = nn.Sequential(
@@ -535,7 +609,8 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         self,
         base_action: Tensor,
         pooled_context: Tensor,
-    ) -> tuple[Tensor, Tensor, Tensor]:
+        task_names: list[str],
+    ) -> tuple[Tensor, Tensor, Tensor, list[list[str]]]:
         batch_size = pooled_context.shape[0]
         mode_logits = self.proposal_mode_head(pooled_context)
         mode_residuals = []
@@ -553,6 +628,13 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         )
         proposal_candidates = []
         proposal_logits = []
+        proposal_mode_names = [
+            [
+                proposal_mode_vocab(task_name, self.config.num_proposal_modes)[int(mode_assignments[slot_idx])]
+                for slot_idx in range(self.config.num_candidates)
+            ]
+            for task_name in task_names
+        ]
         for slot_idx in range(self.config.num_candidates):
             mode_idx = int(mode_assignments[slot_idx])
             candidate = base_action + 0.35 * torch.tanh(mode_residuals[:, mode_idx]) + 0.05 * torch.tanh(slot_deltas[slot_idx]).unsqueeze(0)
@@ -571,7 +653,7 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         stacked_candidates = torch.stack(proposal_candidates, dim=1)
         stacked_logits = torch.stack(proposal_logits, dim=1)
         stacked_candidates[:, 0] = base_action
-        return stacked_candidates, stacked_logits, mode_logits
+        return stacked_candidates, stacked_logits, mode_logits, proposal_mode_names
 
     def forward(
         self,
@@ -581,6 +663,7 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         reveal_tokens: Tensor | None = None,
         memory_token: Tensor | None = None,
         compute_equivariance_probe: bool = False,
+        task_names: list[str] | None = None,
     ) -> dict[str, Tensor]:
         if memory_tokens is None:
             memory_tokens = memory_token
@@ -602,6 +685,20 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
         if memory_tokens is not None:
             decoder_memory = torch.cat([decoder_memory, memory_tokens], dim=1)
 
+        canonical_task_names = [infer_task_name_from_text(name) for name in (task_names or ["generic"] * batch_size)]
+        task_ids = torch.as_tensor(
+            [TASK_INDEX[name] for name in canonical_task_names if name in TASK_INDEX],
+            device=scene_tokens.device,
+            dtype=torch.long,
+        )
+        if task_ids.numel() != batch_size:
+            task_ids = torch.as_tensor(
+                [TASK_INDEX.get(name, 0) for name in canonical_task_names],
+                device=scene_tokens.device,
+                dtype=torch.long,
+            )
+        interaction_context = interaction_context + self.task_embedding(task_ids)
+
         base_queries = self.query_embed.weight.unsqueeze(0).expand(batch_size, -1, -1)
         arm_mean, arm_log_std, coordination = self._decode_arm_tokens(
             queries=base_queries,
@@ -621,7 +718,11 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
             ],
             dim=-1,
         )
-        proposal_candidates, proposal_logits, proposal_mode_logits = self._proposal_outputs(action_mean, pooled_context)
+        proposal_candidates, proposal_logits, proposal_mode_logits, proposal_mode_names = self._proposal_outputs(
+            action_mean,
+            pooled_context,
+            canonical_task_names,
+        )
 
         outputs = {
             "decoded_tokens": torch.cat([arm_mean[:, 0], arm_mean[:, 1]], dim=-1),
@@ -637,7 +738,8 @@ class SymmetricCoordinatedChunkDecoder(nn.Module):
                 self.config.num_candidates,
                 device=scene_tokens.device,
             ) % self.config.num_proposal_modes,
-            "proposal_mode_names": list(DEFAULT_PROPOSAL_MODES[: self.config.num_proposal_modes]),
+            "proposal_mode_names": proposal_mode_names,
+            "proposal_task_names": canonical_task_names,
         }
         if compute_equivariance_probe:
             swapped_phase, swapped_roles, swapped_context = self._conditioning(

code/reveal_vla_bimanual/models/backbones.py

@@ -10,6 +10,8 @@ import torch
 import torch.nn.functional as F
 from torch import Tensor, nn
 
+from pytorch3d.transforms import matrix_to_quaternion
+
 
 @dataclass
 class FrozenVLBackboneConfig:
@@ -22,6 +24,9 @@ class FrozenVLBackboneConfig:
     depth_patch_size: int = 16
     geometry_feature_dim: int = 8
     use_camera_geometry: bool = True
+    use_depth_tokens: bool = True
+    use_geometry_tokens: bool = True
+    use_camera_pose_tokens: bool = True
 
 
 class DepthPatchAdapter(nn.Module):
@@ -65,63 +70,64 @@ class DepthPatchAdapter(nn.Module):
         batch_views, _, height, width = depths.shape
         grid_h = max(1, height // self.patch_size)
         grid_w = max(1, width // self.patch_size)
-        y_coords = torch.linspace(-1.0, 1.0, steps=grid_h, device=depths.device, dtype=depths.dtype)
-        x_coords = torch.linspace(-1.0, 1.0, steps=grid_w, device=depths.device, dtype=depths.dtype)
-        grid_y, grid_x = torch.meshgrid(y_coords, x_coords, indexing="ij")
-        coords = torch.stack([grid_x, grid_y], dim=-1).reshape(1, grid_h * grid_w, 2).expand(batch_views, -1, -1)
+        patch_center_y = torch.linspace(
+            self.patch_size * 0.5,
+            max(self.patch_size * 0.5, height - (self.patch_size * 0.5)),
+            steps=grid_h,
+            device=depths.device,
+            dtype=depths.dtype,
+        )
+        patch_center_x = torch.linspace(
+            self.patch_size * 0.5,
+            max(self.patch_size * 0.5, width - (self.patch_size * 0.5)),
+            steps=grid_w,
+            device=depths.device,
+            dtype=depths.dtype,
+        )
+        pixel_y, pixel_x = torch.meshgrid(patch_center_y, patch_center_x, indexing="ij")
+        norm_x = ((pixel_x / max(width - 1, 1)) * 2.0 - 1.0).reshape(1, grid_h * grid_w, 1)
+        norm_y = ((pixel_y / max(height - 1, 1)) * 2.0 - 1.0).reshape(1, grid_h * grid_w, 1)
+        coords = torch.cat([norm_x, norm_y], dim=-1).expand(batch_views, -1, -1)
 
-        geometry_terms: list[Tensor] = [coords]
         if camera_intrinsics is not None:
-            fx = camera_intrinsics[:, 0, 0].unsqueeze(-1).unsqueeze(-1)
-            fy = camera_intrinsics[:, 1, 1].unsqueeze(-1).unsqueeze(-1)
-            cx = camera_intrinsics[:, 0, 2].unsqueeze(-1).unsqueeze(-1)
-            cy = camera_intrinsics[:, 1, 2].unsqueeze(-1).unsqueeze(-1)
-            intrinsic_features = torch.cat(
-                [
-                    fx.expand(-1, grid_h * grid_w, -1),
-                    fy.expand(-1, grid_h * grid_w, -1),
-                    cx.expand(-1, grid_h * grid_w, -1),
-                    cy.expand(-1, grid_h * grid_w, -1),
-                ],
-                dim=-1,
-            )
-            geometry_terms.append(intrinsic_features)
+            fx = camera_intrinsics[:, 0, 0].unsqueeze(-1)
+            fy = camera_intrinsics[:, 1, 1].unsqueeze(-1)
+            cx = camera_intrinsics[:, 0, 2].unsqueeze(-1)
+            cy = camera_intrinsics[:, 1, 2].unsqueeze(-1)
+            patch_x = pixel_x.reshape(1, grid_h * grid_w).expand(batch_views, -1)
+            patch_y = pixel_y.reshape(1, grid_h * grid_w).expand(batch_views, -1)
+            ray_x = (patch_x - cx) / fx.clamp_min(1e-6)
+            ray_y = (patch_y - cy) / fy.clamp_min(1e-6)
         else:
-            geometry_terms.append(torch.zeros(batch_views, grid_h * grid_w, 4, device=depths.device, dtype=depths.dtype))
+            ray_x = coords[..., 0]
+            ray_y = coords[..., 1]
+        ray_camera = torch.stack([ray_x, ray_y, torch.ones_like(ray_x)], dim=-1)
+        ray_camera = F.normalize(ray_camera, dim=-1)
 
         if camera_extrinsics is not None:
-            translation = camera_extrinsics[:, :3, 3]
-            translation = translation.unsqueeze(1).expand(-1, grid_h * grid_w, -1)
-            geometry_terms.append(translation)
+            rotation = camera_extrinsics[:, :3, :3]
+            translation = camera_extrinsics[:, :3, 3].unsqueeze(1).expand(-1, grid_h * grid_w, -1)
+            ray_world = torch.matmul(rotation, ray_camera.transpose(1, 2)).transpose(1, 2)
+            quaternion = matrix_to_quaternion(rotation)
         else:
-            geometry_terms.append(torch.zeros(batch_views, grid_h * grid_w, 3, device=depths.device, dtype=depths.dtype))
+            rotation = None
+            translation = torch.zeros(batch_views, grid_h * grid_w, 3, device=depths.device, dtype=depths.dtype)
+            ray_world = ray_camera
+            quaternion = torch.zeros(batch_views, 4, device=depths.device, dtype=depths.dtype)
+            quaternion[:, 0] = 1.0
 
-        geometry = torch.cat(geometry_terms, dim=-1)
+        geometry = torch.cat([coords, ray_world, translation], dim=-1)
         if geometry.shape[-1] < self.geometry_feature_dim:
             pad = self.geometry_feature_dim - geometry.shape[-1]
             geometry = F.pad(geometry, (0, pad))
         elif geometry.shape[-1] > self.geometry_feature_dim:
             geometry = geometry[..., : self.geometry_feature_dim]
 
-        if camera_intrinsics is not None:
-            camera_summary = torch.cat(
-                [
-                    camera_intrinsics[:, 0, 0:1],
-                    camera_intrinsics[:, 1, 1:2],
-                    camera_intrinsics[:, 0, 2:3],
-                    camera_intrinsics[:, 1, 2:3],
-                ],
-                dim=-1,
-            )
-        else:
-            camera_summary = torch.zeros(batch_views, 4, device=depths.device, dtype=depths.dtype)
         if camera_extrinsics is not None:
-            camera_summary = torch.cat([camera_summary, camera_extrinsics[:, :3, 3]], dim=-1)
+            translation_summary = camera_extrinsics[:, :3, 3]
         else:
-            camera_summary = torch.cat(
-                [camera_summary, torch.zeros(batch_views, 3, device=depths.device, dtype=depths.dtype)],
-                dim=-1,
-            )
+            translation_summary = torch.zeros(batch_views, 3, device=depths.device, dtype=depths.dtype)
+        camera_summary = torch.cat([quaternion, translation_summary], dim=-1)
         return geometry, camera_summary
 
     def forward(
@@ -130,6 +136,8 @@ class DepthPatchAdapter(nn.Module):
         depth_valid: Tensor | None = None,
         camera_intrinsics: Tensor | None = None,
         camera_extrinsics: Tensor | None = None,
+        include_geometry_features: bool = True,
+        include_camera_pose: bool = True,
     ) -> dict[str, Tensor]:
         if depths.ndim == 4:
             depths = depths.unsqueeze(2)
@@ -161,7 +169,12 @@ class DepthPatchAdapter(nn.Module):
             camera_intrinsics=flat_intrinsics,
             camera_extrinsics=flat_extrinsics,
         )
-        token_inputs = torch.cat([depth_patch, valid_patch, geometry_features], dim=-1)
+        if not include_geometry_features:
+            geometry_features = torch.zeros_like(geometry_features)
+        if not include_camera_pose:
+            camera_summary = torch.zeros_like(camera_summary)
+        # Keep depth tokens depth-only so depth, geometry, and pose ablations are separable.
+        token_inputs = torch.cat([depth_patch, valid_patch, torch.zeros_like(geometry_features)], dim=-1)
         depth_tokens = self.depth_proj(token_inputs)
         geometry_tokens = self.geometry_proj(geometry_features)
         camera_tokens = self.camera_proj(camera_summary).unsqueeze(1)
@@ -324,12 +337,21 @@ class FrozenVLBackbone(nn.Module):
         camera_intrinsics: Tensor | None = None,
         camera_extrinsics: Tensor | None = None,
         return_aux: bool = False,
+        use_depth_tokens: bool | None = None,
+        use_geometry_tokens: bool | None = None,
+        use_camera_pose_tokens: bool | None = None,
     ) -> Tensor | dict[str, Tensor | None]:
         rgb_tokens = self._encode_rgb_tokens(images)
         wants_aux = return_aux or depths is not None or depth_valid is not None or camera_intrinsics is not None or camera_extrinsics is not None
         if not wants_aux:
             return rgb_tokens
 
+        depth_enabled = self.config.use_depth_tokens if use_depth_tokens is None else use_depth_tokens
+        geometry_enabled = self.config.use_geometry_tokens if use_geometry_tokens is None else use_geometry_tokens
+        camera_pose_enabled = self.config.use_camera_pose_tokens if use_camera_pose_tokens is None else use_camera_pose_tokens
+        geometry_enabled = bool(self.config.use_camera_geometry and geometry_enabled)
+        camera_pose_enabled = bool(self.config.use_camera_geometry and camera_pose_enabled)
+
         depth_outputs: dict[str, Tensor | None] = {
             "depth_tokens": None,
             "geometry_tokens": None,
@@ -341,7 +363,15 @@ class FrozenVLBackbone(nn.Module):
                 depth_valid=depth_valid,
                 camera_intrinsics=camera_intrinsics,
                 camera_extrinsics=camera_extrinsics,
+                include_geometry_features=geometry_enabled,
+                include_camera_pose=camera_pose_enabled,
             )
+            if not depth_enabled:
+                depth_outputs["depth_tokens"] = None
+            if not geometry_enabled:
+                depth_outputs["geometry_tokens"] = None
+            if not camera_pose_enabled:
+                depth_outputs["camera_tokens"] = None
 
         return {
             "rgb_tokens": rgb_tokens,

code/reveal_vla_bimanual/models/multiview_fusion.py

@@ -83,6 +83,7 @@ class MultiViewFusion(nn.Module):
         proprio: Tensor,
         language_tokens: Tensor,
         depth_tokens: Tensor | None = None,
+        geometry_tokens: Tensor | None = None,
         camera_tokens: Tensor | None = None,
         return_aux: bool = False,
     ) -> Tensor | dict[str, Tensor]:
@@ -102,6 +103,8 @@ class MultiViewFusion(nn.Module):
             geometry_sources = []
             if depth_tokens is not None:
                 geometry_sources.append(depth_tokens[:, view_idx])
+            if geometry_tokens is not None:
+                geometry_sources.append(geometry_tokens[:, view_idx])
             if camera_tokens is not None:
                 geometry_sources.append(camera_tokens[:, view_idx])
             if geometry_sources:
@@ -119,7 +122,12 @@ class MultiViewFusion(nn.Module):
             batch_size, self.config.proprio_tokens, hidden_dim
         )
         scene_tokens = torch.cat([fused, proprio_tokens, language_tokens], dim=1)
-        if not (return_aux or depth_tokens is not None or camera_tokens is not None):
+        if not (
+            return_aux
+            or depth_tokens is not None
+            or geometry_tokens is not None
+            or camera_tokens is not None
+        ):
             return scene_tokens
         return {
             "scene_tokens": scene_tokens,

code/reveal_vla_bimanual/models/observation_memory.py

@@ -234,6 +234,14 @@ class _SelectiveMemoryBank(nn.Module):
             nn.GELU(),
         )
 
+    def _recency_weights(self, length: int, device: torch.device, dtype: torch.dtype) -> Tensor:
+        if length <= 0:
+            return torch.zeros((0,), device=device, dtype=dtype)
+        positions = torch.arange(length, device=device, dtype=dtype)
+        distances = (length - 1) - positions
+        weights = torch.exp(-0.5 * distances)
+        return weights / weights.sum().clamp_min(1e-6)
+
     def _truncate(self, history: Tensor | None) -> Tensor | None:
         if history is None or history.numel() == 0:
             return history
@@ -241,21 +249,48 @@ class _SelectiveMemoryBank(nn.Module):
             return history
         return history[:, -self.history_steps :]
 
+    def _chunk_pool(self, tokens: Tensor) -> Tensor:
+        batch_size, seq_len, hidden_dim = tokens.shape
+        chunk_size = max(1, (seq_len + self.bank_queries.shape[0] - 1) // self.bank_queries.shape[0])
+        slots = []
+        for slot_idx in range(self.bank_queries.shape[0]):
+            start = slot_idx * chunk_size
+            end = min(seq_len, start + chunk_size)
+            if start >= seq_len:
+                pooled = tokens[:, -1]
+            else:
+                pooled = tokens[:, start:end].mean(dim=1)
+            slots.append(pooled)
+        return torch.stack(slots, dim=1)
+
+    def _compress_tokens(self, tokens: Tensor) -> Tensor:
+        base_slots = self._chunk_pool(tokens)
+        queries = self.bank_queries.unsqueeze(0).expand(tokens.shape[0], -1, -1) + base_slots
+        attended, _ = self.bank_attention(queries, tokens, tokens)
+        return base_slots + 0.1 * attended
+
     def forward(
         self,
-        pooled_current: Tensor,
+        current_tokens: Tensor,
         history_scene_tokens: Tensor | None = None,
         history_actions: Tensor | None = None,
     ) -> dict[str, Tensor]:
         history_scene_tokens = self._truncate(history_scene_tokens)
-        pooled_current = pooled_current.unsqueeze(1)
+        current_bank = self._compress_tokens(current_tokens)
+        pooled_current = current_bank.mean(dim=1, keepdim=True)
         if history_scene_tokens is not None and history_scene_tokens.numel() > 0:
-            history_pooled = history_scene_tokens.mean(dim=2)
+            batch_size, history_steps = history_scene_tokens.shape[:2]
+            flat_history = history_scene_tokens.reshape(batch_size * history_steps, history_scene_tokens.shape[2], history_scene_tokens.shape[3])
+            history_bank = self._compress_tokens(flat_history).view(batch_size, history_steps, self.bank_queries.shape[0], self.hidden_dim)
+            history_pooled = history_bank.mean(dim=2)
             if history_actions is not None and history_actions.numel() > 0:
                 history_actions = history_actions[:, -history_pooled.shape[1] :]
-                history_pooled = history_pooled + self.action_proj(history_actions)
+                history_action_tokens = self.action_proj(history_actions).unsqueeze(2)
+                history_bank = history_bank + history_action_tokens
+                history_pooled = history_bank.mean(dim=2)
             sequence = torch.cat([history_pooled, pooled_current], dim=1)
         else:
+            history_bank = current_bank.unsqueeze(1)[:, :0]
             history_pooled = pooled_current[:, :0]
             sequence = pooled_current
         if sequence.shape[1] > self.position_embedding.shape[1]:
@@ -264,17 +299,21 @@ class _SelectiveMemoryBank(nn.Module):
             )
         encoded = self.sequence_encoder(sequence + self.position_embedding[:, : sequence.shape[1]])
         current_token = encoded[:, -1]
-        prior_token = encoded[:, :-1].mean(dim=1) if encoded.shape[1] > 1 else torch.zeros_like(current_token)
-        novelty = torch.abs(current_token - prior_token)
-        informative = novelty.mean(dim=-1, keepdim=True)
-        gate_logit = self.write_gate(torch.cat([current_token, prior_token, novelty], dim=-1))
-        gate = torch.sigmoid(gate_logit)
-        gate = gate * (informative > (self.write_threshold - self.suppression_margin)).to(gate.dtype)
-        recent_summary = encoded[:, -min(max(1, self.bank_queries.shape[0]), encoded.shape[1]) :].mean(dim=1, keepdim=True)
-        queries = self.bank_queries.unsqueeze(0).expand(encoded.shape[0], -1, -1) + recent_summary
-        bank_tokens, _ = self.bank_attention(queries, encoded, encoded)
-        bank_tokens = bank_tokens + recent_summary
-        bank_tokens = prior_token.unsqueeze(1) * (1.0 - gate.unsqueeze(1)) + bank_tokens * gate.unsqueeze(1)
+        if history_bank.shape[1] > 0:
+            recency = self._recency_weights(
+                history_bank.shape[1],
+                device=history_bank.device,
+                dtype=history_bank.dtype,
+            ).view(1, -1, 1, 1)
+            prior_bank = (history_bank * recency).sum(dim=1)
+        else:
+            prior_bank = torch.zeros_like(current_bank)
+        novelty = torch.abs(current_bank - prior_bank)
+        gate_logit = self.write_gate(torch.cat([current_bank, prior_bank, novelty], dim=-1))
+        novelty_score = novelty.mean(dim=-1, keepdim=True)
+        novelty_gate = torch.sigmoid(12.0 * (novelty_score - self.write_threshold))
+        gate = (0.25 + 0.75 * torch.sigmoid(gate_logit)) * novelty_gate
+        bank_tokens = prior_bank * (1.0 - gate) + current_bank * gate
         bank_tokens = self.token_proj(bank_tokens)
         return {
             "memory_tokens": bank_tokens,
@@ -332,14 +371,13 @@ class DualObservationMemory(nn.Module):
         history_scene_tokens: Tensor | None = None,
         history_actions: Tensor | None = None,
     ) -> dict[str, Tensor]:
-        pooled_current = scene_tokens.mean(dim=1)
         scene_output = self.scene_memory(
-            pooled_current=pooled_current,
+            current_tokens=scene_tokens,
             history_scene_tokens=history_scene_tokens,
             history_actions=history_actions,
         )
         belief_output = self.belief_memory(
-            pooled_current=pooled_current,
+            current_tokens=scene_tokens,
             history_scene_tokens=history_scene_tokens,
             history_actions=history_actions,
         )

code/reveal_vla_bimanual/models/planner.py

@@ -250,10 +250,18 @@ class StructuredElasticUtility(nn.Module):
         occluder_contact = self._field_mean(rollout_state["occluder_contact_field"]).mean(dim=-1)
         grasp_affordance = self._field_mean(rollout_state["grasp_affordance_field"]).mean(dim=-1)
         support_stability = torch.sigmoid(self._field_mean(rollout_state["support_stability_field"])).mean(dim=-1)
-        persistence = self._field_mean(rollout_state["persistence_field"]).mean(dim=-1)
-        reocclusion = self._field_mean(rollout_state["reocclusion_field"]).mean(dim=-1)
-        disturbance = self._field_mean(rollout_state["disturbance_field"]).mean(dim=-1)
-        access_quality = torch.sigmoid(self._field_mean(rollout_state["access_field"])).mean(dim=-1)
+        persistence_traj = self._field_mean(rollout_state["persistence_field"])
+        reocclusion_traj = self._field_mean(rollout_state["reocclusion_field"])
+        disturbance_traj = self._field_mean(rollout_state["disturbance_field"])
+        access_traj = torch.sigmoid(self._field_mean(rollout_state["access_field"]))
+        persistence = persistence_traj.mean(dim=-1)
+        reocclusion = reocclusion_traj.mean(dim=-1)
+        disturbance = disturbance_traj.mean(dim=-1)
+        access_quality = access_traj.mean(dim=-1)
+        access_floor = access_traj.amin(dim=-1)
+        persistence_floor = persistence_traj.amin(dim=-1)
+        support_floor = torch.sigmoid(self._field_mean(rollout_state["support_stability_field"])).amin(dim=-1)
+        reocclusion_worst = reocclusion_traj.amax(dim=-1)
         retrieve_progress = torch.sigmoid(candidate_chunks[:, :, :, -1]).mean(dim=-1)
         utility = (
             self.config.belief_gain_weight * (belief_future - initial_belief)
@@ -278,8 +286,12 @@ class StructuredElasticUtility(nn.Module):
             "persistence": persistence,
             "support_stability": support_stability,
             "reocclusion_penalty": reocclusion,
+            "reocclusion_worst": reocclusion_worst,
             "disturbance_penalty": disturbance,
             "access_quality": access_quality,
+            "access_floor": access_floor,
+            "persistence_floor": persistence_floor,
+            "support_floor": support_floor,
             "task_progress": retrieve_progress,
             "utility_structured": utility,
         }
@@ -346,14 +358,42 @@ class CascadePlanner(nn.Module):
         self.structured = StructuredElasticUtility(config)
         self.residual = ResidualPlannerScorer(config)
 
-    def shortlist(self, proposal_logits: Tensor | None, candidate_chunks: Tensor) -> Tensor:
+    def shortlist(
+        self,
+        proposal_logits: Tensor | None,
+        candidate_chunks: Tensor,
+        proposal_mode_assignments: Tensor | None = None,
+    ) -> Tensor:
         batch_size, num_candidates = candidate_chunks.shape[:2]
         top_k = min(max(1, self.config.top_k), num_candidates)
         if proposal_logits is None:
             cheap_scores = -candidate_chunks.square().mean(dim=(-1, -2))
         else:
             cheap_scores = proposal_logits
-        return cheap_scores.topk(top_k, dim=-1).indices
+        if proposal_mode_assignments is None:
+            return cheap_scores.topk(top_k, dim=-1).indices
+        if proposal_mode_assignments.ndim == 1:
+            proposal_mode_assignments = proposal_mode_assignments.unsqueeze(0).expand(batch_size, -1)
+
+        shortlisted = []
+        for batch_idx in range(batch_size):
+            scores = cheap_scores[batch_idx]
+            mode_ids = proposal_mode_assignments[batch_idx]
+            mode_best: list[tuple[float, int]] = []
+            for mode_id in torch.unique(mode_ids):
+                mode_indices = torch.nonzero(mode_ids == mode_id, as_tuple=False).squeeze(-1)
+                best_local = mode_indices[scores[mode_indices].argmax()]
+                mode_best.append((float(scores[best_local]), int(best_local)))
+            mode_best.sort(key=lambda item: item[0], reverse=True)
+            chosen = [index for _, index in mode_best[:top_k]]
+            if len(chosen) < top_k:
+                for candidate_idx in scores.argsort(descending=True).tolist():
+                    if candidate_idx not in chosen:
+                        chosen.append(candidate_idx)
+                    if len(chosen) >= top_k:
+                        break
+            shortlisted.append(torch.as_tensor(chosen[:top_k], device=candidate_chunks.device, dtype=torch.long))
+        return torch.stack(shortlisted, dim=0)
 
     def select_best(
         self,
@@ -362,6 +402,7 @@ class CascadePlanner(nn.Module):
         rollout_state: dict[str, Tensor],
         proposal_logits: Tensor | None = None,
         candidate_indices: Tensor | None = None,
+        proposal_mode_names: list[list[str]] | None = None,
     ) -> dict[str, Tensor]:
         structured = self.structured(
             initial_state=initial_state,
@@ -375,6 +416,22 @@ class CascadePlanner(nn.Module):
         )
         utility_total = structured["utility_structured"] + self.config.residual_weight * residual["utility_residual"]
         utility_total = utility_total + residual["success_logits"].sigmoid() - residual["risk_values"]
+        feasibility_penalty = torch.zeros_like(utility_total)
+        if proposal_mode_names is not None:
+            retrieve_like = torch.zeros_like(utility_total, dtype=torch.bool)
+            for batch_idx, names in enumerate(proposal_mode_names):
+                for candidate_idx, name in enumerate(names[: utility_total.shape[1]]):
+                    retrieve_like[batch_idx, candidate_idx] = any(
+                        token in name for token in ("retrieve", "insert_actor", "probe_inside")
+                    )
+            blocked = (
+                (structured["access_floor"] < 0.15)
+                | (structured["persistence_floor"] < 0.15)
+                | (structured["support_floor"] < 0.25)
+                | (structured["reocclusion_worst"] > 0.6)
+            )
+            feasibility_penalty = retrieve_like.to(dtype=utility_total.dtype) * blocked.to(dtype=utility_total.dtype) * 2.0
+            utility_total = utility_total - feasibility_penalty
         best_local = utility_total.argmax(dim=-1)
         batch_indices = torch.arange(candidate_chunks.shape[0], device=candidate_chunks.device)
         if candidate_indices is None:
@@ -386,6 +443,7 @@ class CascadePlanner(nn.Module):
             **residual,
             "utility_total": utility_total,
             "utility_scores": utility_total,
+            "feasibility_penalty": feasibility_penalty,
             "best_indices": best_indices,
             "best_chunk": candidate_chunks[batch_indices, best_local],
             "ranking_diagnostics": {

code/reveal_vla_bimanual/models/policy.py

@@ -11,6 +11,7 @@ from models.action_decoder import (
     ChunkDecoderConfig,
     InteractionChunkDecoder,
     SymmetricCoordinatedChunkDecoder,
+    infer_task_name_from_text,
 )
 from models.backbones import FrozenVLBackbone, FrozenVLBackboneConfig
 from models.multiview_fusion import MultiViewFusion, MultiViewFusionConfig
@@ -65,6 +66,11 @@ class BackboneOnlyPolicy(nn.Module):
             attention_mask=language_tokens["attention_mask"],
         )
 
+    def _task_names(self, batch_size: int, texts: Sequence[str] | None = None) -> list[str]:
+        if texts is None:
+            return ["generic"] * batch_size
+        return [infer_task_name_from_text(text) for text in texts]
+
     def encode_scene(
         self,
         images: Tensor,
@@ -388,6 +394,8 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
         camera_intrinsics: Tensor | None = None,
         camera_extrinsics: Tensor | None = None,
         use_depth: bool = True,
+        use_geometry_tokens: bool | None = None,
+        use_camera_pose_tokens: bool | None = None,
     ) -> dict[str, Tensor]:
         encoded = self.backbone.encode_images(
             images,
@@ -396,6 +404,9 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             camera_intrinsics=camera_intrinsics if use_depth else None,
             camera_extrinsics=camera_extrinsics if use_depth else None,
             return_aux=True,
+            use_depth_tokens=use_depth,
+            use_geometry_tokens=use_geometry_tokens,
+            use_camera_pose_tokens=use_camera_pose_tokens,
         )
         assert isinstance(encoded, dict)
         text_tokens = self._encode_language(images, texts=texts, language_tokens=language_tokens)
@@ -404,6 +415,7 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             proprio=proprio,
             language_tokens=text_tokens,
             depth_tokens=encoded.get("depth_tokens"),
+            geometry_tokens=encoded.get("geometry_tokens"),
             camera_tokens=encoded.get("camera_tokens"),
             return_aux=True,
         )
@@ -413,6 +425,7 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             "view_summaries": fused["view_summaries"],
             "geometry_summaries": fused["geometry_summaries"],
             "depth_tokens": encoded.get("depth_tokens"),
+            "geometry_tokens": encoded.get("geometry_tokens"),
             "camera_tokens": encoded.get("camera_tokens"),
         }
 
@@ -441,6 +454,8 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
         camera_intrinsics: Tensor | None = None,
         camera_extrinsics: Tensor | None = None,
         use_depth: bool = True,
+        use_geometry_tokens: bool | None = None,
+        use_camera_pose_tokens: bool | None = None,
     ) -> Tensor | None:
         if history_images is None or history_proprio is None or history_images.numel() == 0:
             return None
@@ -469,6 +484,8 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             camera_intrinsics=None,
             camera_extrinsics=None,
             use_depth=use_depth,
+            use_geometry_tokens=use_geometry_tokens,
+            use_camera_pose_tokens=use_camera_pose_tokens,
         )["scene_tokens"]
         return history_scene.view(batch_size, history_steps, history_scene.shape[1], history_scene.shape[2])
 
@@ -495,6 +512,27 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
         value = value.detach()
         return value.unsqueeze(1).unsqueeze(2).expand(-1, num_candidates, horizon, *value.shape[1:])
 
+    def _zero_memory_output(self, scene_tokens: Tensor) -> dict[str, Tensor]:
+        batch_size, _, hidden_dim = scene_tokens.shape
+        scene_memory_tokens = scene_tokens.new_zeros((batch_size, self.config.memory.scene_bank_size, hidden_dim))
+        belief_memory_tokens = scene_tokens.new_zeros((batch_size, self.config.memory.belief_bank_size, hidden_dim))
+        memory_tokens = torch.cat([scene_memory_tokens, belief_memory_tokens], dim=1)
+        return {
+            "scene_memory_tokens": scene_memory_tokens,
+            "belief_memory_tokens": belief_memory_tokens,
+            "memory_tokens": memory_tokens,
+            "memory_token": memory_tokens.mean(dim=1, keepdim=True),
+            "memory_sequence": scene_tokens.new_zeros((batch_size, 0, hidden_dim)),
+            "memory_state": scene_tokens.new_zeros((batch_size, hidden_dim * 2)),
+            "memory_uncertainty": scene_tokens.new_zeros((batch_size,)),
+            "memory_write_rate": scene_tokens.new_zeros((batch_size,)),
+            "memory_saturation": scene_tokens.new_zeros((batch_size,)),
+            "scene_write_gate": scene_tokens.new_zeros((batch_size, self.config.memory.scene_bank_size)),
+            "belief_write_gate": scene_tokens.new_zeros((batch_size, self.config.memory.belief_bank_size)),
+            "memory_scene_state": scene_tokens.new_zeros((batch_size, hidden_dim)),
+            "memory_belief_state": scene_tokens.new_zeros((batch_size, hidden_dim)),
+        }
+
     def _identity_rollout(
         self,
         interaction_state: dict[str, Tensor],
@@ -531,7 +569,14 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
         history_depths: Tensor | None = None,
         history_depth_valid: Tensor | None = None,
         compute_equivariance_probe: bool = False,
+        use_geometry_tokens: bool | None = None,
+        use_camera_pose_tokens: bool | None = None,
+        use_memory: bool = True,
+        use_task_conditioning: bool = True,
+        rollout_mode_override: str | None = None,
+        use_proposal_candidates: bool = True,
     ) -> dict[str, Tensor]:
+        task_names = self._task_names(images.shape[0], texts=texts)
         scene_output = self._encode_scene_with_optional_depth(
             images=images,
             proprio=proprio,
@@ -542,6 +587,8 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             camera_intrinsics=camera_intrinsics,
             camera_extrinsics=camera_extrinsics,
             use_depth=use_depth,
+            use_geometry_tokens=use_geometry_tokens,
+            use_camera_pose_tokens=use_camera_pose_tokens,
         )
         scene_tokens = scene_output["scene_tokens"]
         history_scene_tokens = self.encode_history_with_optional_depth(
@@ -554,19 +601,26 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             camera_intrinsics=camera_intrinsics,
             camera_extrinsics=camera_extrinsics,
             use_depth=use_depth,
+            use_geometry_tokens=use_geometry_tokens,
+            use_camera_pose_tokens=use_camera_pose_tokens,
         )
         if history_steps_override is not None and history_scene_tokens is not None and history_scene_tokens.numel() > 0:
             history_scene_tokens = history_scene_tokens[:, -history_steps_override:]
             if history_actions is not None and history_actions.numel() > 0:
                 history_actions = history_actions[:, -history_steps_override:]
-        memory_output = self.memory(
-            scene_tokens,
-            history_scene_tokens=history_scene_tokens,
-            history_actions=history_actions,
-        )
+        if use_memory:
+            memory_output = self.memory(
+                scene_tokens,
+                history_scene_tokens=history_scene_tokens,
+                history_actions=history_actions,
+            )
+        else:
+            memory_output = self._zero_memory_output(scene_tokens)
         elastic_state = self.elastic_state_head(
             scene_tokens,
             memory_tokens=memory_output["memory_tokens"],
+            task_names=task_names,
+            use_task_conditioning=use_task_conditioning,
         )
         elastic_state["memory_tokens"] = memory_output["memory_tokens"]
         elastic_state["memory_token"] = memory_output["memory_token"]
@@ -581,6 +635,7 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             interaction_state=elastic_state,
             memory_tokens=memory_output["memory_tokens"],
             compute_equivariance_probe=compute_equivariance_probe,
+            task_names=task_names,
         )
         outputs = {
             **decoded,
@@ -592,7 +647,11 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             "reveal_state": elastic_state,
             "view_summaries": scene_output["view_summaries"],
             "geometry_summaries": scene_output["geometry_summaries"],
+            "depth_tokens": scene_output["depth_tokens"],
+            "geometry_tokens": scene_output["geometry_tokens"],
+            "camera_tokens": scene_output["camera_tokens"],
             "rollout_source": "none",
+            "task_names": task_names,
         }
 
         candidate_chunks = candidate_chunks_override
@@ -602,8 +661,10 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
                 outputs["action_mean"],
                 outputs["action_log_std"],
                 num_candidates=self.config.decoder.num_candidates,
-                proposal_candidates=outputs.get("proposal_candidates"),
+                proposal_candidates=outputs.get("proposal_candidates") if use_proposal_candidates else None,
             )
+            if not use_proposal_candidates:
+                proposal_logits = None
         else:
             proposal_logits = None
         outputs["candidate_chunks"] = candidate_chunks
@@ -625,13 +686,25 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             )
             return outputs
 
-        shortlist_indices = self.planner.shortlist(proposal_logits=proposal_logits, candidate_chunks=candidate_chunks)
+        shortlist_indices = self.planner.shortlist(
+            proposal_logits=proposal_logits,
+            candidate_chunks=candidate_chunks,
+            proposal_mode_assignments=outputs.get("proposal_mode_assignments") if use_proposal_candidates else None,
+        )
         outputs["planner_topk_indices"] = shortlist_indices
         batch_size = candidate_chunks.shape[0]
         batch_indices = torch.arange(batch_size, device=candidate_chunks.device).unsqueeze(-1)
         topk_candidates = candidate_chunks[batch_indices, shortlist_indices]
         num_topk = topk_candidates.shape[1]
         outputs["planner_topk_candidates"] = topk_candidates
+        proposal_mode_names = outputs.get("proposal_mode_names")
+        topk_proposal_mode_names = None
+        if proposal_mode_names is not None and use_proposal_candidates:
+            topk_proposal_mode_names = [
+                [proposal_mode_names[batch_idx][int(candidate_idx.item())] for candidate_idx in shortlist_indices[batch_idx]]
+                for batch_idx in range(batch_size)
+            ]
+            outputs["planner_topk_mode_names"] = topk_proposal_mode_names
         if proposal_logits is not None:
             topk_proposal_logits = proposal_logits.gather(1, shortlist_indices)
         else:
@@ -653,6 +726,7 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
                 rollout_state=identity_rollout,
                 proposal_logits=topk_proposal_logits,
                 candidate_indices=shortlist_indices,
+                proposal_mode_names=topk_proposal_mode_names,
             )
             outputs["planned_rollout"] = identity_rollout
             outputs["planned_chunk"] = selected["best_chunk"]
@@ -677,6 +751,8 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             memory_tokens=self._tile_tensor(memory_output["memory_tokens"], num_topk),
             scene_memory_tokens=self._tile_tensor(memory_output["scene_memory_tokens"], num_topk),
             belief_memory_tokens=self._tile_tensor(memory_output["belief_memory_tokens"], num_topk),
+            task_names=[name for name in task_names for _ in range(num_topk)],
+            rollout_mode_override=rollout_mode_override,
         )
         reshaped_rollout = {
             key: value.view(batch_size, num_topk, *value.shape[1:]) for key, value in rollout.items()
@@ -687,6 +763,7 @@ class ElasticRevealBimanualPolicy(BackboneOnlyPolicy):
             rollout_state=reshaped_rollout,
             proposal_logits=topk_proposal_logits,
             candidate_indices=shortlist_indices,
+            proposal_mode_names=topk_proposal_mode_names,
         )
         outputs["planned_rollout"] = reshaped_rollout
         outputs["planned_chunk"] = selected["best_chunk"]

code/reveal_vla_bimanual/models/reveal_head.py

@@ -1,12 +1,103 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
+from typing import Sequence
 
 import torch
 import torch.nn.functional as F
 from torch import Tensor, nn
 
 
+HEAD_TASKS = ("generic", "foliage", "bag", "cloth")
+TASK_METRIC_NAMES = (
+    "opening_quality",
+    "actor_feasibility_score",
+    "gap_width",
+    "damage_proxy",
+    "release_collapse_rate",
+    "target_visibility_confidence",
+    "mouth_aperture",
+    "hold_quality",
+    "rim_slip_risk",
+    "insertable_actor_corridor",
+    "layer_separation_quality",
+    "fold_preservation",
+    "insertion_corridor",
+    "top_layer_stability",
+    "lift_too_much_risk",
+)
+TASK_INDEX = {name: idx for idx, name in enumerate(HEAD_TASKS)}
+
+
+def task_ids_from_names(task_names: Sequence[str] | None, device: torch.device, batch_size: int) -> Tensor:
+    if task_names is None:
+        return torch.zeros(batch_size, device=device, dtype=torch.long)
+    return torch.as_tensor(
+        [TASK_INDEX.get(str(name), 0) for name in task_names],
+        device=device,
+        dtype=torch.long,
+    )
+
+
+def _mean_map(value: Tensor) -> Tensor:
+    return value.mean(dim=(-1, -2)).squeeze(1)
+
+
+def compute_task_metrics_from_fields(
+    *,
+    access_field: Tensor,
+    persistence_field: Tensor,
+    disturbance_field: Tensor,
+    reocclusion_field: Tensor,
+    visibility_field: Tensor,
+    clearance_field: Tensor,
+    support_stability_field: Tensor,
+    uncertainty_field: Tensor,
+) -> dict[str, Tensor]:
+    access_prob = torch.sigmoid(access_field)
+    opening_mask = access_prob.amax(dim=1, keepdim=True)
+    support_stability = torch.sigmoid(support_stability_field)
+    visibility_prob = torch.sigmoid(visibility_field)
+    clearance_prob = torch.sigmoid(clearance_field).mean(dim=1, keepdim=True)
+    normalized_uncertainty = uncertainty_field / (1.0 + uncertainty_field)
+
+    opening_quality_field = opening_mask * persistence_field * support_stability
+    newly_revealed_field = torch.relu(visibility_prob - reocclusion_field)
+    still_visible_field = visibility_prob * persistence_field
+    reoccluded_field = reocclusion_field
+
+    opening_quality = _mean_map(opening_quality_field)
+    actor_feasibility_score = 0.6 * _mean_map(clearance_prob) + 0.4 * _mean_map(opening_mask)
+    base_gap = _mean_map(opening_mask)
+    disturbance_cost = _mean_map(disturbance_field)
+    support_quality = _mean_map(support_stability)
+    visibility_confidence = _mean_map(visibility_prob * (1.0 - normalized_uncertainty))
+    reocclusion_rate = _mean_map(reocclusion_field)
+    persistence_score = _mean_map(persistence_field)
+
+    return {
+        "newly_revealed_field": newly_revealed_field,
+        "still_visible_field": still_visible_field,
+        "reoccluded_field": reoccluded_field,
+        "opening_quality_field": opening_quality_field,
+        "opening_quality": torch.clamp(opening_quality, 0.0, 1.0),
+        "actor_feasibility_score": torch.clamp(actor_feasibility_score, 0.0, 1.0),
+        "gap_width": 0.03 + 0.21 * torch.clamp(base_gap, 0.0, 1.0),
+        "damage_proxy": torch.clamp(disturbance_cost + 0.5 * (1.0 - support_quality), 0.0, 1.0),
+        "release_collapse_rate": torch.clamp(reocclusion_rate, 0.0, 1.0),
+        "target_visibility_confidence": torch.clamp(visibility_confidence, 0.0, 1.0),
+        "mouth_aperture": torch.clamp(base_gap, 0.0, 1.0),
+        "hold_quality": torch.clamp(0.5 * (persistence_score + support_quality), 0.0, 1.0),
+        "rim_slip_risk": torch.clamp(reocclusion_rate + 0.5 * (1.0 - support_quality), 0.0, 1.0),
+        "insertable_actor_corridor": torch.clamp(0.6 * actor_feasibility_score + 0.4 * base_gap, 0.0, 1.0),
+        "layer_separation_quality": torch.clamp(0.7 * base_gap + 0.3 * actor_feasibility_score, 0.0, 1.0),
+        "fold_preservation": torch.clamp(1.0 - disturbance_cost, 0.0, 1.0),
+        "insertion_corridor": torch.clamp(0.5 * actor_feasibility_score + 0.5 * base_gap, 0.0, 1.0),
+        "top_layer_stability": torch.clamp(support_quality, 0.0, 1.0),
+        "lift_too_much_risk": torch.clamp(disturbance_cost + 0.5 * torch.relu(base_gap - 0.5), 0.0, 1.0),
+    }
+
+
 @dataclass
 class RevealHeadConfig:
     hidden_dim: int = 512
@@ -20,6 +111,7 @@ class RevealHeadConfig:
     num_phases: int = 5
     num_arm_roles: int = 4
     num_interaction_tokens: int = 8
+    num_tasks: int = len(HEAD_TASKS)
 
 
 class RevealStateHead(nn.Module):
@@ -379,6 +471,22 @@ class ElasticOcclusionFieldDecoder(nn.Module):
             nn.GELU(),
             nn.Linear(config.hidden_dim, config.num_support_modes),
         )
+        self.task_embedding = nn.Embedding(config.num_tasks, config.hidden_dim)
+        self.task_field_affine = nn.Linear(config.hidden_dim, config.hidden_dim * 2)
+        self.task_summary_adapter = nn.Sequential(
+            nn.LayerNorm(config.hidden_dim * 2),
+            nn.Linear(config.hidden_dim * 2, config.hidden_dim),
+            nn.GELU(),
+        )
+        self.task_phase_head = nn.Linear(config.hidden_dim, config.num_phases)
+        self.task_support_head = nn.Linear(config.hidden_dim, config.num_support_modes)
+        self.task_reocclusion_head = nn.Linear(config.hidden_dim, config.num_support_modes)
+        self.task_metric_head = nn.Sequential(
+            nn.LayerNorm(config.hidden_dim * 2),
+            nn.Linear(config.hidden_dim * 2, config.hidden_dim),
+            nn.GELU(),
+            nn.Linear(config.hidden_dim, len(TASK_METRIC_NAMES)),
+        )
 
     def _pool_source(self, source_tokens: Tensor | None, fallback: Tensor) -> Tensor:
         if source_tokens is None or source_tokens.numel() == 0:
@@ -403,6 +511,8 @@ class ElasticOcclusionFieldDecoder(nn.Module):
         interaction_tokens: Tensor,
         scene_tokens: Tensor | None = None,
         memory_tokens: Tensor | None = None,
+        task_names: Sequence[str] | None = None,
+        use_task_conditioning: bool = True,
     ) -> dict[str, Tensor]:
         batch_size = interaction_tokens.shape[0]
         pooled_interaction = interaction_tokens.mean(dim=1)
@@ -423,6 +533,15 @@ class ElasticOcclusionFieldDecoder(nn.Module):
         pooled_field = field_tokens.mean(dim=1)
         summary_input = torch.cat([pooled_interaction, pooled_field, pooled_scene, pooled_memory], dim=-1)
         latent_summary = self.summary_proj(summary_input)
+        task_ids = task_ids_from_names(task_names, interaction_tokens.device, batch_size)
+        task_embed = self.task_embedding(task_ids)
+        if use_task_conditioning:
+            scale, bias = self.task_field_affine(task_embed).chunk(2, dim=-1)
+            grid = grid * (1.0 + 0.1 * scale.view(batch_size, self.config.hidden_dim, 1, 1))
+            grid = grid + 0.1 * bias.view(batch_size, self.config.hidden_dim, 1, 1)
+            task_summary = latent_summary + 0.1 * self.task_summary_adapter(torch.cat([latent_summary, task_embed], dim=-1))
+        else:
+            task_summary = latent_summary
 
         access_field = self.access_field(grid)
         target_belief_field = self.target_belief_field(grid)
@@ -435,6 +554,23 @@ class ElasticOcclusionFieldDecoder(nn.Module):
         reocclusion_field = torch.sigmoid(self.reocclusion_field(grid))
         disturbance_field = torch.sigmoid(self.disturbance_field(grid))
         uncertainty_field = F.softplus(self.uncertainty_field(grid))
+        task_metrics = compute_task_metrics_from_fields(
+            access_field=access_field,
+            persistence_field=persistence_field,
+            disturbance_field=disturbance_field,
+            reocclusion_field=reocclusion_field,
+            visibility_field=visibility_field,
+            clearance_field=clearance_field,
+            support_stability_field=support_stability_field,
+            uncertainty_field=uncertainty_field,
+        )
+        metric_residuals = 0.05 * torch.tanh(
+            self.task_metric_head(torch.cat([task_summary, task_embed], dim=-1))
+        )
+        metric_residual_map = {
+            name: metric_residuals[:, idx]
+            for idx, name in enumerate(TASK_METRIC_NAMES)
+        }
 
         support_stability_prob = torch.sigmoid(support_stability_field)
         risk_field = torch.sigmoid(
@@ -459,7 +595,7 @@ class ElasticOcclusionFieldDecoder(nn.Module):
         arm_identity = self.arm_identity.weight.unsqueeze(0).expand(batch_size, -1, -1)
         arm_tokens = pooled_interaction.unsqueeze(1).expand(-1, 2, -1) + arm_identity
         arm_role_input = torch.cat(
-            [arm_tokens, latent_summary.unsqueeze(1).expand(-1, 2, -1)],
+            [arm_tokens, task_summary.unsqueeze(1).expand(-1, 2, -1)],
             dim=-1,
         )
         arm_role_logits = self.arm_role_head(arm_role_input)
@@ -477,8 +613,8 @@ class ElasticOcclusionFieldDecoder(nn.Module):
             risk_field.mean(dim=(-1, -2)).squeeze(1),
             uncertainty_field.mean(dim=(-1, -2)).squeeze(1),
             access_prob.mean(dim=(-1, -2)).transpose(0, 1).transpose(0, 1),
-            self.support_mode(summary_input),
-            self.phase_head(summary_input),
+            self.support_mode(summary_input) + (self.task_support_head(task_summary) if use_task_conditioning else 0.0),
+            self.phase_head(summary_input) + (self.task_phase_head(task_summary) if use_task_conditioning else 0.0),
             arm_role_logits.reshape(batch_size, -1),
         ]
         compact_state = torch.cat(
@@ -487,7 +623,7 @@ class ElasticOcclusionFieldDecoder(nn.Module):
         )
 
         output = {
-            "phase_logits": self.phase_head(summary_input),
+            "phase_logits": self.phase_head(summary_input) + (self.task_phase_head(task_summary) if use_task_conditioning else 0.0),
             "arm_role_logits": arm_role_logits,
             "target_belief_field": target_belief_field,
             "visibility_field": visibility_field,
@@ -502,20 +638,34 @@ class ElasticOcclusionFieldDecoder(nn.Module):
             "uncertainty_field": uncertainty_field,
             "interaction_tokens": interaction_tokens,
             "field_tokens": field_tokens,
-            "latent_summary": latent_summary,
-            "support_mode_logits": self.support_mode(summary_input),
+            "latent_summary": task_summary,
+            "support_mode_logits": self.support_mode(summary_input) + (self.task_support_head(task_summary) if use_task_conditioning else 0.0),
             "corridor_logits": corridor_logits,
             "persistence_horizon": persistence_horizon,
             "disturbance_cost": disturbance_cost,
             "belief_map": target_belief_map,
-            "reocclusion_logit": self.reocclusion_head(summary_input),
+            "reocclusion_logit": self.reocclusion_head(summary_input) + (self.task_reocclusion_head(task_summary) if use_task_conditioning else 0.0),
             "persistence_uncertainty": uncertainty_field.mean(dim=(-1, -2)).squeeze(1),
             "access_field": access_field,
             "uncertainty": uncertainty_field.mean(dim=(-1, -2)).squeeze(1),
             "compact_state": compact_state,
+            "task_ids": task_ids,
         }
         output["target_field"] = output["target_belief_field"]
         output["actor_feasibility_field"] = output["clearance_field"]
+        output.update(
+            {
+                "newly_revealed_field": task_metrics["newly_revealed_field"],
+                "still_visible_field": task_metrics["still_visible_field"],
+                "reoccluded_field": task_metrics["reoccluded_field"],
+                "opening_quality_field": task_metrics["opening_quality_field"],
+            }
+        )
+        for name in TASK_METRIC_NAMES:
+            if name == "gap_width":
+                output[name] = torch.clamp(task_metrics[name] + 0.01 * metric_residual_map[name], 0.0, 1.0)
+            else:
+                output[name] = torch.clamp(task_metrics[name] + metric_residual_map[name], 0.0, 1.0)
         return output
 
 
@@ -544,6 +694,8 @@ class ElasticOcclusionStateHead(nn.Module):
         scene_tokens: Tensor,
         memory_token: Tensor | None = None,
         memory_tokens: Tensor | None = None,
+        task_names: Sequence[str] | None = None,
+        use_task_conditioning: bool = True,
     ) -> dict[str, Tensor]:
         if memory_tokens is None:
             memory_tokens = memory_token
@@ -558,4 +710,6 @@ class ElasticOcclusionStateHead(nn.Module):
             interaction_tokens=interaction_tokens,
             scene_tokens=scene_tokens,
             memory_tokens=memory_tokens,
+            task_names=task_names,
+            use_task_conditioning=use_task_conditioning,
         )

code/reveal_vla_bimanual/models/world_model.py

@@ -1,11 +1,13 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
+from typing import Sequence
 
 import torch
+import torch.nn.functional as F
 from torch import Tensor, nn
 
-from models.reveal_head import InteractionFieldDecoder
+from models.reveal_head import InteractionFieldDecoder, compute_task_metrics_from_fields, task_ids_from_names
 
 
 @dataclass
@@ -24,6 +26,8 @@ class RevealWMConfig:
     predict_belief_map: bool = True
     scene_bank_size: int = 2
     belief_bank_size: int = 2
+    rollout_mode: str = "spatial_rollout"
+    num_tasks: int = 4
 
 
 class RevealWM(nn.Module):
@@ -167,6 +171,7 @@ class ElasticOcclusionWorldModel(nn.Module):
             + config.num_phases
             + (2 * config.num_arm_roles)
         )
+        self.compact_state_dim = compact_state_dim
         self.state_encoder = nn.Sequential(
             nn.LayerNorm(compact_state_dim),
             nn.Linear(compact_state_dim, config.hidden_dim),
@@ -203,6 +208,43 @@ class ElasticOcclusionWorldModel(nn.Module):
         self.disturbance_head = nn.Linear(config.hidden_dim, field_elements)
         self.uncertainty_head = nn.Linear(config.hidden_dim, field_elements)
         self.access_head = nn.Linear(config.hidden_dim, config.num_support_modes * field_elements)
+        field_channels = 12 + config.num_support_modes
+        spatial_hidden = max(32, config.hidden_dim // 2)
+        self.task_embedding = nn.Embedding(config.num_tasks, config.hidden_dim)
+        self.spatial_field_encoder = nn.Sequential(
+            nn.Conv2d(field_channels, spatial_hidden, kernel_size=3, padding=1),
+            nn.GELU(),
+            nn.Conv2d(spatial_hidden, config.hidden_dim, kernel_size=3, padding=1),
+            nn.GELU(),
+        )
+        self.spatial_context_proj = nn.Sequential(
+            nn.LayerNorm(config.hidden_dim * 4),
+            nn.Linear(config.hidden_dim * 4, config.hidden_dim),
+            nn.GELU(),
+        )
+        self.spatial_gate_z = nn.Conv2d(config.hidden_dim * 2, config.hidden_dim, kernel_size=3, padding=1)
+        self.spatial_gate_r = nn.Conv2d(config.hidden_dim * 2, config.hidden_dim, kernel_size=3, padding=1)
+        self.spatial_candidate = nn.Conv2d(config.hidden_dim * 2, config.hidden_dim, kernel_size=3, padding=1)
+        self.spatial_summary_proj = nn.Sequential(
+            nn.LayerNorm(config.hidden_dim * 3),
+            nn.Linear(config.hidden_dim * 3, config.hidden_dim),
+            nn.GELU(),
+        )
+        self.spatial_phase_head = nn.Linear(config.hidden_dim, config.num_phases)
+        self.spatial_support_mode_head = nn.Linear(config.hidden_dim, config.num_support_modes)
+        self.spatial_arm_role_head = nn.Linear(config.hidden_dim, 2 * config.num_arm_roles)
+        self.spatial_reocclusion_head = nn.Linear(config.hidden_dim, config.num_support_modes)
+        self.spatial_target_belief_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_visibility_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_clearance_head = nn.Conv2d(config.hidden_dim, 2, kernel_size=1)
+        self.spatial_occluder_contact_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_grasp_affordance_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_support_stability_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_persistence_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_reocclusion_field_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_disturbance_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_uncertainty_head = nn.Conv2d(config.hidden_dim, 1, kernel_size=1)
+        self.spatial_access_head = nn.Conv2d(config.hidden_dim, config.num_support_modes, kernel_size=1)
 
     def _compact_from_state(self, interaction_state: dict[str, Tensor]) -> Tensor:
         if "compact_state" in interaction_state:
@@ -226,6 +268,32 @@ class ElasticOcclusionWorldModel(nn.Module):
         ]
         return torch.cat([component if component.ndim > 1 else component.unsqueeze(-1) for component in components], dim=-1)
 
+    def _repeat_state_rollout(self, interaction_state: dict[str, Tensor], horizon: int) -> dict[str, Tensor]:
+        rollout: dict[str, Tensor] = {}
+        for key, value in interaction_state.items():
+            if isinstance(value, Tensor):
+                rollout[key] = value.unsqueeze(1).expand(-1, horizon, *value.shape[1:])
+        return rollout
+
+    def _stack_state_fields(self, interaction_state: dict[str, Tensor]) -> Tensor:
+        return torch.cat(
+            [
+                interaction_state["target_belief_field"],
+                interaction_state["visibility_field"],
+                interaction_state["clearance_field"],
+                interaction_state["occluder_contact_field"],
+                interaction_state["grasp_affordance_field"],
+                interaction_state["support_stability_field"],
+                interaction_state["persistence_field"],
+                interaction_state["reocclusion_field"],
+                interaction_state["disturbance_field"],
+                interaction_state["risk_field"],
+                interaction_state["uncertainty_field"],
+                interaction_state["access_field"],
+            ],
+            dim=1,
+        )
+
     def _decode_fields(self, latent: Tensor) -> dict[str, Tensor]:
         batch_size = latent.shape[0]
         side = self.config.field_size
@@ -257,7 +325,7 @@ class ElasticOcclusionWorldModel(nn.Module):
         weighted_persistence = (persistence_field.expand_as(access_prob) * access_prob).sum(dim=(-1, -2))
         access_mass = access_prob.sum(dim=(-1, -2)).clamp_min(1e-4)
         persistence_horizon = self.config.rollout_horizon * weighted_persistence / access_mass
-        return {
+        outputs = {
             "target_belief_field": target_belief_field,
             "visibility_field": visibility_field,
             "clearance_field": clearance_field,
@@ -282,6 +350,96 @@ class ElasticOcclusionWorldModel(nn.Module):
             "target_field": target_belief_field,
             "actor_feasibility_field": clearance_field,
         }
+        outputs.update(
+            compute_task_metrics_from_fields(
+                access_field=access_field,
+                persistence_field=persistence_field,
+                disturbance_field=disturbance_field,
+                reocclusion_field=reocclusion_field,
+                visibility_field=visibility_field,
+                clearance_field=clearance_field,
+                support_stability_field=support_stability_field,
+                uncertainty_field=uncertainty_field,
+            )
+        )
+        return outputs
+
+    def _decode_spatial_fields(self, hidden: Tensor, summary: Tensor) -> dict[str, Tensor]:
+        target_belief_field = self.spatial_target_belief_head(hidden)
+        visibility_field = self.spatial_visibility_head(hidden)
+        clearance_field = self.spatial_clearance_head(hidden)
+        occluder_contact_field = self.spatial_occluder_contact_head(hidden)
+        grasp_affordance_field = self.spatial_grasp_affordance_head(hidden)
+        support_stability_field = self.spatial_support_stability_head(hidden)
+        persistence_field = torch.sigmoid(self.spatial_persistence_head(hidden))
+        reocclusion_field = torch.sigmoid(self.spatial_reocclusion_field_head(hidden))
+        disturbance_field = torch.sigmoid(self.spatial_disturbance_head(hidden))
+        uncertainty_field = F.softplus(self.spatial_uncertainty_head(hidden))
+        access_field = self.spatial_access_head(hidden)
+        support_stability_prob = torch.sigmoid(support_stability_field)
+        risk_field = torch.sigmoid(
+            disturbance_field
+            + 0.75 * reocclusion_field
+            + 0.5 * (1.0 - support_stability_prob)
+            + 0.25 * uncertainty_field
+        )
+        corridor_source = access_field.amax(dim=-2)
+        corridor_logits = F.interpolate(
+            corridor_source,
+            size=self.config.num_approach_templates,
+            mode="linear",
+            align_corners=False,
+        )
+        access_prob = torch.sigmoid(access_field)
+        weighted_persistence = (persistence_field.expand_as(access_prob) * access_prob).sum(dim=(-1, -2))
+        access_mass = access_prob.sum(dim=(-1, -2)).clamp_min(1e-4)
+        persistence_horizon = self.config.rollout_horizon * weighted_persistence / access_mass
+        compact_state = self.compact_decoder(summary)
+        role_slice = self.spatial_arm_role_head(summary).view(summary.shape[0], 2, self.config.num_arm_roles)
+        outputs = {
+            "target_belief_field": target_belief_field,
+            "visibility_field": visibility_field,
+            "clearance_field": clearance_field,
+            "occluder_contact_field": occluder_contact_field,
+            "grasp_affordance_field": grasp_affordance_field,
+            "support_stability_field": support_stability_field,
+            "persistence_field": persistence_field,
+            "reocclusion_field": reocclusion_field,
+            "disturbance_field": disturbance_field,
+            "risk_field": risk_field,
+            "uncertainty_field": uncertainty_field,
+            "access_field": access_field,
+            "corridor_logits": corridor_logits,
+            "persistence_horizon": persistence_horizon,
+            "disturbance_cost": disturbance_field.mean(dim=(-1, -2)).squeeze(1),
+            "belief_map": F.interpolate(
+                target_belief_field,
+                size=(self.config.belief_map_size, self.config.belief_map_size),
+                mode="bilinear",
+                align_corners=False,
+            ),
+            "target_field": target_belief_field,
+            "actor_feasibility_field": clearance_field,
+            "compact_state": compact_state,
+            "phase_logits": self.spatial_phase_head(summary),
+            "arm_role_logits": role_slice,
+            "support_mode_logits": self.spatial_support_mode_head(summary),
+            "reocclusion_logit": self.spatial_reocclusion_head(summary),
+            "uncertainty": uncertainty_field.mean(dim=(-1, -2)).squeeze(1),
+        }
+        outputs.update(
+            compute_task_metrics_from_fields(
+                access_field=access_field,
+                persistence_field=persistence_field,
+                disturbance_field=disturbance_field,
+                reocclusion_field=reocclusion_field,
+                visibility_field=visibility_field,
+                clearance_field=clearance_field,
+                support_stability_field=support_stability_field,
+                uncertainty_field=uncertainty_field,
+            )
+        )
+        return outputs
 
     def forward(
         self,
@@ -291,6 +449,8 @@ class ElasticOcclusionWorldModel(nn.Module):
         memory_tokens: Tensor | None = None,
         scene_memory_tokens: Tensor | None = None,
         belief_memory_tokens: Tensor | None = None,
+        task_names: Sequence[str] | None = None,
+        rollout_mode_override: str | None = None,
     ) -> dict[str, Tensor]:
         if scene_memory_tokens is None:
             scene_memory_tokens = interaction_state.get("scene_memory_tokens")
@@ -305,14 +465,57 @@ class ElasticOcclusionWorldModel(nn.Module):
         if belief_memory_tokens is None:
             belief_memory_tokens = scene_tokens[:, :1]
 
-        latent = self.state_encoder(self._compact_from_state(interaction_state))
+        rollout_mode = rollout_mode_override or self.config.rollout_mode
+        horizon = action_chunk.shape[1]
+        if rollout_mode in {"null_rollout", "identity_rollout"}:
+            repeated = self._repeat_state_rollout(interaction_state, horizon)
+            if "scene_memory_tokens" not in repeated:
+                repeated["scene_memory_tokens"] = scene_memory_tokens.unsqueeze(1).expand(-1, horizon, *scene_memory_tokens.shape[1:])
+            if "belief_memory_tokens" not in repeated:
+                repeated["belief_memory_tokens"] = belief_memory_tokens.unsqueeze(1).expand(-1, horizon, *belief_memory_tokens.shape[1:])
+            if "memory_tokens" not in repeated:
+                repeated["memory_tokens"] = torch.cat(
+                    [repeated["scene_memory_tokens"], repeated["belief_memory_tokens"]],
+                    dim=2,
+                )
+            if "memory_token" not in repeated:
+                repeated["memory_token"] = repeated["memory_tokens"].mean(dim=2, keepdim=True)
+            return repeated
+
+        task_ids = task_ids_from_names(task_names, scene_tokens.device, scene_tokens.shape[0])
+        task_embed = self.task_embedding(task_ids)
+        latent = self.state_encoder(self._compact_from_state(interaction_state)) + 0.1 * task_embed
         scene_memory = self.scene_memory_proj(scene_memory_tokens.mean(dim=1))
         belief_memory = self.belief_memory_proj(belief_memory_tokens.mean(dim=1))
         outputs: dict[str, list[Tensor]] = {}
         scene_bias = scene_tokens.mean(dim=1)
 
+        if rollout_mode == "spatial_rollout":
+            hidden = self.spatial_field_encoder(self._stack_state_fields(interaction_state))
+            spatial_context = self.spatial_context_proj(torch.cat([scene_bias, scene_memory, belief_memory, task_embed], dim=-1))
+            hidden = hidden + spatial_context.unsqueeze(-1).unsqueeze(-1)
+            for step in range(horizon):
+                action_latent = self.action_encoder(action_chunk[:, step]) + 0.1 * task_embed
+                input_map = (action_latent + spatial_context).unsqueeze(-1).unsqueeze(-1).expand_as(hidden)
+                z = torch.sigmoid(self.spatial_gate_z(torch.cat([hidden, input_map], dim=1)))
+                r = torch.sigmoid(self.spatial_gate_r(torch.cat([hidden, input_map], dim=1)))
+                candidate = torch.tanh(self.spatial_candidate(torch.cat([r * hidden, input_map], dim=1)))
+                hidden = (1.0 - z) * hidden + z * candidate
+                pooled_hidden = hidden.mean(dim=(-1, -2))
+                scene_memory = 0.75 * scene_memory + 0.25 * torch.tanh(self.scene_memory_update(pooled_hidden))
+                belief_memory = 0.65 * belief_memory + 0.35 * torch.tanh(self.belief_memory_update(pooled_hidden))
+                summary = self.spatial_summary_proj(torch.cat([pooled_hidden, scene_bias, task_embed], dim=-1))
+                decoded = self._decode_spatial_fields(hidden, summary)
+                decoded["scene_memory_tokens"] = scene_memory.unsqueeze(1).expand(-1, self.config.scene_bank_size, -1)
+                decoded["belief_memory_tokens"] = belief_memory.unsqueeze(1).expand(-1, self.config.belief_bank_size, -1)
+                decoded["memory_tokens"] = torch.cat([decoded["scene_memory_tokens"], decoded["belief_memory_tokens"]], dim=1)
+                decoded["memory_token"] = decoded["memory_tokens"].mean(dim=1, keepdim=True)
+                for key, value in decoded.items():
+                    outputs.setdefault(key, []).append(value)
+            return {key: torch.stack(values, dim=1) for key, values in outputs.items()}
+
         for step in range(action_chunk.shape[1]):
-            action_latent = self.action_encoder(action_chunk[:, step])
+            action_latent = self.action_encoder(action_chunk[:, step]) + 0.1 * task_embed
             transition_input = torch.cat([latent, action_latent, scene_memory, belief_memory], dim=-1)
             latent = self.transition(transition_input, latent + 0.1 * scene_bias)
             scene_memory = 0.75 * scene_memory + 0.25 * torch.tanh(self.scene_memory_update(latent))

code/reveal_vla_bimanual/scripts/run_rlbench_handoff_eval.sh

@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="${ROOT_DIR:-/workspace}"
+PROJECT_DIR="${ROOT_DIR}/reveal_vla_bimanual"
+PYTHON_BIN="${PYTHON_BIN:-${ROOT_DIR}/envs/rlbench/bin/python}"
+OUTPUT_ROOT="${OUTPUT_ROOT:-${ROOT_DIR}/reports/rlbench_handoff_matrix}"
+EPISODES_PER_TASK="${EPISODES_PER_TASK:-1}"
+EPISODE_LENGTH="${EPISODE_LENGTH:-20}"
+RESOLUTION="${RESOLUTION:-224}"
+CHUNK_COMMIT_STEPS="${CHUNK_COMMIT_STEPS:-4}"
+BASELINE_CHECKPOINT="${BASELINE_CHECKPOINT:-${ROOT_DIR}/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt}"
+SPATIAL_CHECKPOINT="${SPATIAL_CHECKPOINT:-${ROOT_DIR}/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/checkpoint_best.pt}"
+
+source "${ROOT_DIR}/VLAarchtests_work/environment/runtime_env_vars.sh"
+
+run_sweep() {
+  local output_dir="$1"
+  shift
+  mkdir -p "${output_dir}"
+  (
+    cd "${PROJECT_DIR}"
+    PYTHONPATH="${PROJECT_DIR}" "${PYTHON_BIN}" -m eval.run_peract2_task_sweep "$@"
+  )
+}
+
+mkdir -p "${OUTPUT_ROOT}"
+
+run_sweep \
+  "${OUTPUT_ROOT}/baseline" \
+  --checkpoint "${BASELINE_CHECKPOINT}" \
+  --output-root "${OUTPUT_ROOT}/baseline" \
+  --run-name-prefix baseline_rgbd_seed17 \
+  --episodes-per-task "${EPISODES_PER_TASK}" \
+  --episode-length "${EPISODE_LENGTH}" \
+  --resolution "${RESOLUTION}" \
+  --chunk-commit-steps "${CHUNK_COMMIT_STEPS}" \
+  --allow-unsupervised-planning \
+  --headless \
+  --skip-noplan
+
+run_sweep \
+  "${OUTPUT_ROOT}/spatial_full" \
+  --checkpoint "${SPATIAL_CHECKPOINT}" \
+  --output-root "${OUTPUT_ROOT}/spatial_full" \
+  --run-name-prefix spatial_phase_seed17 \
+  --episodes-per-task "${EPISODES_PER_TASK}" \
+  --episode-length "${EPISODE_LENGTH}" \
+  --resolution "${RESOLUTION}" \
+  --chunk-commit-steps "${CHUNK_COMMIT_STEPS}" \
+  --allow-unsupervised-planning \
+  --headless
+
+run_sweep \
+  "${OUTPUT_ROOT}/spatial_nogeom" \
+  --checkpoint "${SPATIAL_CHECKPOINT}" \
+  --output-root "${OUTPUT_ROOT}/spatial_nogeom" \
+  --run-name-prefix spatial_phase_nogeom_seed17 \
+  --episodes-per-task "${EPISODES_PER_TASK}" \
+  --episode-length "${EPISODE_LENGTH}" \
+  --resolution "${RESOLUTION}" \
+  --chunk-commit-steps "${CHUNK_COMMIT_STEPS}" \
+  --allow-unsupervised-planning \
+  --headless \
+  --no-geometry \
+  --skip-noplan
+
+run_sweep \
+  "${OUTPUT_ROOT}/spatial_compactwm" \
+  --checkpoint "${SPATIAL_CHECKPOINT}" \
+  --output-root "${OUTPUT_ROOT}/spatial_compactwm" \
+  --run-name-prefix spatial_phase_compactwm_seed17 \
+  --episodes-per-task "${EPISODES_PER_TASK}" \
+  --episode-length "${EPISODE_LENGTH}" \
+  --resolution "${RESOLUTION}" \
+  --chunk-commit-steps "${CHUNK_COMMIT_STEPS}" \
+  --allow-unsupervised-planning \
+  --headless \
+  --compact-world-model \
+  --skip-noplan
+
+run_sweep \
+  "${OUTPUT_ROOT}/spatial_notask" \
+  --checkpoint "${SPATIAL_CHECKPOINT}" \
+  --output-root "${OUTPUT_ROOT}/spatial_notask" \
+  --run-name-prefix spatial_phase_notask_seed17 \
+  --episodes-per-task "${EPISODES_PER_TASK}" \
+  --episode-length "${EPISODE_LENGTH}" \
+  --resolution "${RESOLUTION}" \
+  --chunk-commit-steps "${CHUNK_COMMIT_STEPS}" \
+  --allow-unsupervised-planning \
+  --headless \
+  --disable-task-conditioning \
+  --skip-noplan
+
+(
+  cd "${PROJECT_DIR}"
+  PYTHONPATH="${PROJECT_DIR}" "${PYTHON_BIN}" -m eval.compare_rlbench_sweeps \
+    --reference-label baseline_plan \
+    --output-dir "${OUTPUT_ROOT}/comparison" \
+    --run "baseline_plan=${OUTPUT_ROOT}/baseline/baseline_rgbd_seed17_plan_split/rollout_eval.json" \
+    --run "spatial_noplan=${OUTPUT_ROOT}/spatial_full/spatial_phase_seed17_noplan_split/rollout_eval.json" \
+    --run "spatial_plan=${OUTPUT_ROOT}/spatial_full/spatial_phase_seed17_plan_split/rollout_eval.json" \
+    --run "spatial_nogeom=${OUTPUT_ROOT}/spatial_nogeom/spatial_phase_nogeom_seed17_plan_split/rollout_eval.json" \
+    --run "spatial_compactwm=${OUTPUT_ROOT}/spatial_compactwm/spatial_phase_compactwm_seed17_plan_split/rollout_eval.json" \
+    --run "spatial_notask=${OUTPUT_ROOT}/spatial_notask/spatial_phase_notask_seed17_plan_split/rollout_eval.json"
+)

code/reveal_vla_bimanual/sim_reveal/dataset.py

@@ -26,6 +26,11 @@ LEGACY_PRIVILEGED_RENDER_KEYS = frozenset(
 )
 
 
+def dataset_uses_rgbd(dataset_version: Any) -> bool:
+    version = str(dataset_version or "")
+    return version.startswith(RGBD_PROXY_DATASET_VERSION)
+
+
 def _assert_noleak_sample(sample: dict[str, Any]) -> None:
     render_state = sample.get("render_state", {})
     leaked_keys = sorted(LEGACY_PRIVILEGED_RENDER_KEYS.intersection(render_state))
@@ -102,6 +107,8 @@ def collect_teacher_dataset(
                         "language_goal": observation["text"],
                         "action_chunk": action_chunk.astype("float32"),
                         "support_mode": int(privileged_state["support_mode"]),
+                        "phase": int(privileged_state.get("phase_label", 0)),
+                        "subgoal_progress": float(privileged_state.get("subgoal_progress", 0.0)),
                         "corridor_feasible": privileged_state["corridor_feasible"].astype("float32"),
                         "persistence_horizon": privileged_state["persistence_horizon"].astype("float32"),
                         "disturbance_cost": float(privileged_state["disturbance_cost"]),
@@ -114,7 +121,21 @@ def collect_teacher_dataset(
                         "support_stability_map": privileged_state["support_stability_map"].astype("float32"),
                         "reocclusion_target": float(privileged_state["reocclusion_target"]),
                         "reocclusion_map": privileged_state["reocclusion_map"].astype("float32"),
+                        "gap_width": float(privileged_state.get("gap_width", 0.0)),
+                        "damage_proxy": float(privileged_state.get("damage_proxy", 0.0)),
+                        "release_collapse_rate": float(privileged_state.get("release_collapse_rate", 0.0)),
+                        "target_visibility_confidence": float(privileged_state.get("target_visibility_confidence", 0.0)),
+                        "mouth_aperture": float(privileged_state.get("mouth_aperture", 0.0)),
+                        "hold_quality": float(privileged_state.get("hold_quality", 0.0)),
+                        "rim_slip_risk": float(privileged_state.get("rim_slip_risk", 0.0)),
+                        "insertable_actor_corridor": float(privileged_state.get("insertable_actor_corridor", 0.0)),
+                        "layer_separation_quality": float(privileged_state.get("layer_separation_quality", 0.0)),
+                        "fold_preservation": float(privileged_state.get("fold_preservation", 0.0)),
+                        "insertion_corridor": float(privileged_state.get("insertion_corridor", 0.0)),
+                        "top_layer_stability": float(privileged_state.get("top_layer_stability", 0.0)),
+                        "lift_too_much_risk": float(privileged_state.get("lift_too_much_risk", 0.0)),
                         "rollout_support_mode": rollout["rollout_support_mode"].astype("int64"),
+                        "rollout_phase": rollout.get("rollout_phase", np.zeros((rollout["rollout_support_mode"].shape[0],), dtype=np.int64)).astype("int64"),
                         "rollout_corridor_feasible": rollout["rollout_corridor_feasible"].astype("float32"),
                         "rollout_persistence_horizon": rollout["rollout_persistence_horizon"].astype("float32"),
                         "rollout_disturbance_cost": rollout["rollout_disturbance_cost"].astype("float32"),
@@ -189,7 +210,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
         return len(self.samples)
 
     def _render_cache_key(self, sample: dict[str, Any], render_state: dict[str, Any]) -> bytes:
-        include_depth = sample.get("dataset_version") == RGBD_PROXY_DATASET_VERSION
+        include_depth = dataset_uses_rgbd(sample.get("dataset_version"))
         return pickle.dumps(
             (sample["proxy_name"], self.resolution, include_depth, render_state),
             protocol=4,
@@ -200,7 +221,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
         cached = self._render_cache.get(cache_key)
         if cached is not None:
             return cached
-        include_depth = sample.get("dataset_version") == RGBD_PROXY_DATASET_VERSION
+        include_depth = dataset_uses_rgbd(sample.get("dataset_version"))
         rendered = render_views_from_state(
             proxy_name=sample["proxy_name"],
             render_state=render_state,
@@ -216,6 +237,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             return cached_item
         sample = self.samples[index]
         _assert_noleak_sample(sample)
+        candidate_count = int(sample.get("candidate_action_chunks", np.zeros((0, 0, 0), dtype=np.float32)).shape[0])
         images = self._render_sample(sample, sample["render_state"])
         history_images = []
         history_depths = []
@@ -232,7 +254,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
                     dim=0,
                 )
             )
-            if sample.get("dataset_version") == RGBD_PROXY_DATASET_VERSION:
+            if dataset_uses_rgbd(sample.get("dataset_version")):
                 history_depths.append(
                     torch.stack(
                         [
@@ -267,7 +289,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             history_stacked = torch.stack(history_images, dim=0).permute(0, 1, 4, 2, 3).float() / 255.0
         else:
             history_stacked = torch.zeros((0, 3, 3, self.resolution, self.resolution), dtype=torch.float32)
-        if sample.get("dataset_version") == RGBD_PROXY_DATASET_VERSION:
+        if dataset_uses_rgbd(sample.get("dataset_version")):
             depths = torch.stack(
                 [
                     torch.from_numpy(images["front_depth"]),
@@ -317,6 +339,8 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             "texts": sample["language_goal"],
             "action_chunk": torch.as_tensor(sample["action_chunk"], dtype=torch.float32),
             "support_mode": torch.as_tensor(sample["support_mode"], dtype=torch.long),
+            "phase": torch.as_tensor(sample.get("phase", 0), dtype=torch.long),
+            "subgoal_progress": torch.as_tensor(sample.get("subgoal_progress", 0.0), dtype=torch.float32),
             "corridor_feasible": torch.as_tensor(sample["corridor_feasible"], dtype=torch.float32),
             "persistence_horizon": torch.as_tensor(sample["persistence_horizon"], dtype=torch.float32),
             "disturbance_cost": torch.as_tensor(sample["disturbance_cost"], dtype=torch.float32),
@@ -329,7 +353,21 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             "support_stability_map": torch.as_tensor(sample.get("support_stability_map", np.zeros((32, 32), dtype=np.float32)), dtype=torch.float32).unsqueeze(0),
             "reocclusion_target": torch.as_tensor(sample.get("reocclusion_target", 0.0), dtype=torch.float32),
             "reocclusion_map": torch.as_tensor(sample.get("reocclusion_map", np.zeros((32, 32), dtype=np.float32)), dtype=torch.float32).unsqueeze(0),
+            "gap_width": torch.as_tensor(sample.get("gap_width", 0.0), dtype=torch.float32),
+            "damage_proxy": torch.as_tensor(sample.get("damage_proxy", 0.0), dtype=torch.float32),
+            "release_collapse_rate": torch.as_tensor(sample.get("release_collapse_rate", 0.0), dtype=torch.float32),
+            "target_visibility_confidence": torch.as_tensor(sample.get("target_visibility_confidence", 0.0), dtype=torch.float32),
+            "mouth_aperture": torch.as_tensor(sample.get("mouth_aperture", 0.0), dtype=torch.float32),
+            "hold_quality": torch.as_tensor(sample.get("hold_quality", 0.0), dtype=torch.float32),
+            "rim_slip_risk": torch.as_tensor(sample.get("rim_slip_risk", 0.0), dtype=torch.float32),
+            "insertable_actor_corridor": torch.as_tensor(sample.get("insertable_actor_corridor", 0.0), dtype=torch.float32),
+            "layer_separation_quality": torch.as_tensor(sample.get("layer_separation_quality", 0.0), dtype=torch.float32),
+            "fold_preservation": torch.as_tensor(sample.get("fold_preservation", 0.0), dtype=torch.float32),
+            "insertion_corridor": torch.as_tensor(sample.get("insertion_corridor", 0.0), dtype=torch.float32),
+            "top_layer_stability": torch.as_tensor(sample.get("top_layer_stability", 0.0), dtype=torch.float32),
+            "lift_too_much_risk": torch.as_tensor(sample.get("lift_too_much_risk", 0.0), dtype=torch.float32),
             "rollout_support_mode": torch.as_tensor(sample["rollout_support_mode"], dtype=torch.long),
+            "rollout_phase": torch.as_tensor(sample.get("rollout_phase", np.zeros((0,), dtype=np.int64)), dtype=torch.long),
             "rollout_corridor_feasible": torch.as_tensor(sample["rollout_corridor_feasible"], dtype=torch.float32),
             "rollout_persistence_horizon": torch.as_tensor(sample["rollout_persistence_horizon"], dtype=torch.float32),
             "rollout_disturbance_cost": torch.as_tensor(sample["rollout_disturbance_cost"], dtype=torch.float32),
@@ -342,6 +380,7 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             "rollout_grasp_affordance_map": torch.as_tensor(sample.get("rollout_grasp_affordance_map", np.zeros((0, 32, 32), dtype=np.float32)), dtype=torch.float32),
             "candidate_action_chunks": torch.as_tensor(sample["candidate_action_chunks"], dtype=torch.float32),
             "candidate_rollout_support_mode": torch.as_tensor(sample["candidate_rollout_support_mode"], dtype=torch.long),
+            "candidate_rollout_phase": torch.as_tensor(sample.get("candidate_rollout_phase", np.zeros((0, 0), dtype=np.int64)), dtype=torch.long),
             "candidate_rollout_corridor_feasible": torch.as_tensor(sample["candidate_rollout_corridor_feasible"], dtype=torch.float32),
             "candidate_rollout_persistence_horizon": torch.as_tensor(sample["candidate_rollout_persistence_horizon"], dtype=torch.float32),
             "candidate_rollout_disturbance_cost": torch.as_tensor(sample["candidate_rollout_disturbance_cost"], dtype=torch.float32),
@@ -356,8 +395,23 @@ class RevealOfflineDataset(Dataset[dict[str, Any]]):
             "candidate_final_disturbance_cost": torch.as_tensor(sample["candidate_final_disturbance_cost"], dtype=torch.float32),
             "candidate_reocclusion_rate": torch.as_tensor(sample["candidate_reocclusion_rate"], dtype=torch.float32),
             "candidate_visibility_integral": torch.as_tensor(sample["candidate_visibility_integral"], dtype=torch.float32),
+            "candidate_actor_feasibility_auc": torch.as_tensor(sample.get("candidate_actor_feasibility_auc", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_reveal_achieved": torch.as_tensor(sample.get("candidate_reveal_achieved", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_hold_persistence": torch.as_tensor(sample.get("candidate_hold_persistence", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_support_stability_auc": torch.as_tensor(sample.get("candidate_support_stability_auc", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_disturbance_auc": torch.as_tensor(sample.get("candidate_disturbance_auc", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_macro_ids": torch.as_tensor(sample.get("candidate_macro_ids", np.zeros((candidate_count,), dtype=np.int64)), dtype=torch.long),
+            "candidate_is_hard_negative": torch.as_tensor(sample.get("candidate_is_hard_negative", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
             "candidate_risk": torch.as_tensor(sample["candidate_risk"], dtype=torch.float32),
             "candidate_utility": torch.as_tensor(sample["candidate_utility"], dtype=torch.float32),
+            "candidate_gap_width": torch.as_tensor(sample.get("candidate_gap_width", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_damage_proxy": torch.as_tensor(sample.get("candidate_damage_proxy", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_mouth_aperture": torch.as_tensor(sample.get("candidate_mouth_aperture", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_hold_quality": torch.as_tensor(sample.get("candidate_hold_quality", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_rim_slip_risk": torch.as_tensor(sample.get("candidate_rim_slip_risk", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_fold_preservation": torch.as_tensor(sample.get("candidate_fold_preservation", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_layer_separation_quality": torch.as_tensor(sample.get("candidate_layer_separation_quality", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
+            "candidate_lift_too_much_risk": torch.as_tensor(sample.get("candidate_lift_too_much_risk", np.zeros((candidate_count,), dtype=np.float32)), dtype=torch.float32),
             "proxy_name": sample["proxy_name"],
             "episode_id": sample["episode_id"],
         }

code/reveal_vla_bimanual/sim_reveal/procedural_envs.py

@@ -347,6 +347,53 @@ class ProceduralRevealEnv:
         horizon_ratio = persistence[current_mode] / float(max(1, self.rollout_horizon))
         return float(np.clip(1.0 - horizon_ratio + 0.35 * self.disturbance, 0.0, 1.0))
 
+    def _phase_label(
+        self,
+        visibility: float,
+        corridor: np.ndarray,
+        persistence: np.ndarray,
+        disturbance_cost: float,
+    ) -> int:
+        support_mode = int(self._current_support_mode())
+        corridor_ready = bool(corridor[support_mode, self.target_template] > 0.5)
+        persistence_ratio = persistence[support_mode] / float(max(1, self.rollout_horizon))
+        opening_ready = self.opening >= (0.75 * self.dynamics.desired_opening)
+        retrieve_ready = (
+            corridor_ready
+            and visibility >= self.dynamics.retrieve_visibility_threshold
+            and self.actor_progress >= 0.55
+        )
+        recovering = disturbance_cost >= 0.55 or (opening_ready and persistence_ratio < 0.35)
+        if retrieve_ready:
+            return 3
+        if recovering:
+            return 4
+        if opening_ready and persistence_ratio >= 0.6:
+            return 2
+        if self.opening < self.dynamics.desired_opening or visibility < self.dynamics.retrieve_visibility_threshold:
+            return 1
+        return 0
+
+    def _subgoal_progress(
+        self,
+        visibility: float,
+        corridor: np.ndarray,
+        persistence: np.ndarray,
+    ) -> float:
+        support_mode = int(self._current_support_mode())
+        corridor_mass = float(corridor[support_mode].mean())
+        persistence_ratio = float(persistence[support_mode] / float(max(1, self.rollout_horizon)))
+        return float(
+            np.clip(
+                0.35 * self.opening
+                + 0.25 * visibility
+                + 0.20 * corridor_mass
+                + 0.20 * persistence_ratio,
+                0.0,
+                1.0,
+            )
+        )
+
     def _grasp_affordance_map(
         self,
         belief_map: np.ndarray,
@@ -374,6 +421,40 @@ class ProceduralRevealEnv:
         reocclusion_target = self._reocclusion_target(persistence)
         reocclusion_map = np.full((32, 32), reocclusion_target, dtype=np.float32)
         grasp_affordance_map = self._grasp_affordance_map(belief_map, visibility_map, clearance_map)
+        task_metrics: dict[str, float] = {}
+        if self.proxy_name == FOLIAGE_PROXY.name:
+            task_metrics = {
+                "gap_width": float(np.clip(0.03 + 0.16 * self.opening, 0.03, 0.24)),
+                "damage_proxy": disturbance_cost,
+                "release_collapse_rate": reocclusion_target,
+                "target_visibility_confidence": visibility,
+            }
+        elif self.proxy_name == BAG_PROXY.name:
+            task_metrics = {
+                "mouth_aperture": float(self.opening),
+                "hold_quality": support_stability,
+                "rim_slip_risk": reocclusion_target,
+                "insertable_actor_corridor": float(corridor[support_mode, self.target_template]),
+            }
+        elif self.proxy_name == CLOTH_PROXY.name:
+            task_metrics = {
+                "layer_separation_quality": float(np.clip(self.opening * (1.0 - 0.20 * self.disturbance), 0.0, 1.0)),
+                "fold_preservation": float(np.clip(1.0 - disturbance_cost, 0.0, 1.0)),
+                "insertion_corridor": float(corridor[support_mode, self.target_template]),
+                "top_layer_stability": support_stability,
+                "lift_too_much_risk": float(np.clip(max(0.0, self.opening - self.dynamics.desired_opening), 0.0, 1.0)),
+            }
+        phase_label = self._phase_label(
+            visibility=visibility,
+            corridor=corridor,
+            persistence=persistence,
+            disturbance_cost=disturbance_cost,
+        )
+        subgoal_progress = self._subgoal_progress(
+            visibility=visibility,
+            corridor=corridor,
+            persistence=persistence,
+        )
         return {
             "support_mode": support_mode,
             "corridor_feasible": corridor,
@@ -391,6 +472,9 @@ class ProceduralRevealEnv:
             "visibility": visibility,
             "retrieval_success": bool(self.retrieved),
             "target_template": self.target_template,
+            "phase_label": int(phase_label),
+            "subgoal_progress": float(subgoal_progress),
+            **task_metrics,
         }
 
     def render_state(self, privileged_state: dict[str, Any] | None = None) -> dict[str, Any]:
@@ -467,6 +551,99 @@ class ProceduralRevealEnv:
         action[13] = np.float32(1.0 if retrieve else -1.0)
         return action
 
+    def _set_mode_bits(self, action: np.ndarray, mode: SupportMode) -> None:
+        action[1] = np.float32(1.0 if mode == SupportMode.TRANSFER else -1.0)
+        action[2] = np.float32(1.0 if mode == SupportMode.PASSIVE else -1.0)
+        action[6] = np.float32(1.0 if mode == SupportMode.HOLD else -1.0)
+
+    def macro_action_chunk(self, macro_name: str, chunk_horizon: int = 8) -> np.ndarray:
+        preferred_mode = self.dynamics.preferred_mode
+        hold_mode = SupportMode.HOLD
+        passive_mode = SupportMode.PASSIVE
+        target_index = self.target_template
+        left_index = max(0, target_index - 4)
+        right_index = min(self.num_templates - 1, target_index + 4)
+        wrong_index = 0 if target_index > (self.num_templates // 2) else self.num_templates - 1
+
+        chunk = np.zeros((chunk_horizon, 14), dtype=np.float32)
+        for step_idx in range(chunk_horizon):
+            action = self.teacher_action()
+            action[13] = np.float32(-1.0)
+            action[8] = np.float32(0.2)
+            self._set_mode_bits(action, preferred_mode)
+
+            if macro_name in {"widen_gap", "widen_mouth", "lift_edge", "separate_layer"}:
+                self._set_mode_bits(action, hold_mode)
+                action[0] = np.float32(0.95)
+            elif macro_name in {"maintain_gap", "maintain_mouth", "maintain_lift", "stabilize_fold", "pin_canopy"}:
+                self._set_mode_bits(action, preferred_mode)
+                action[0] = np.float32(0.12)
+            elif macro_name in {"sweep_left", "pin_left_rim"}:
+                self._set_mode_bits(action, hold_mode)
+                action[0] = np.float32(0.75)
+                action[7] = np.float32(self._normalized_template(left_index))
+            elif macro_name in {"sweep_right", "pin_right_rim"}:
+                self._set_mode_bits(action, hold_mode)
+                action[0] = np.float32(0.75)
+                action[7] = np.float32(self._normalized_template(right_index))
+            elif macro_name == "probe_inside":
+                self._set_mode_bits(action, preferred_mode)
+                action[0] = np.float32(0.10)
+                action[8] = np.float32(0.75)
+            elif macro_name == "insert_actor":
+                self._set_mode_bits(action, preferred_mode)
+                action[0] = np.float32(0.10)
+                action[8] = np.float32(1.0)
+            elif macro_name == "retrieve":
+                self._set_mode_bits(action, preferred_mode)
+                action[0] = np.float32(0.05)
+                action[8] = np.float32(1.0)
+                action[13] = np.float32(1.0)
+            elif macro_name == "premature_retrieve":
+                self._set_mode_bits(action, passive_mode)
+                action[0] = np.float32(-0.20)
+                action[8] = np.float32(1.0)
+                action[13] = np.float32(1.0)
+            elif macro_name in {"reveal_with_release", "foliage_immediate_reocclusion"}:
+                reveal_phase = step_idx < max(1, chunk_horizon // 2)
+                self._set_mode_bits(action, hold_mode if reveal_phase else passive_mode)
+                action[0] = np.float32(0.95 if reveal_phase else -0.35)
+                action[8] = np.float32(0.2)
+            elif macro_name in {"wrong_side_reveal", "wrong_edge_reveal", "wrong_layer_reveal"}:
+                self._set_mode_bits(action, hold_mode)
+                action[0] = np.float32(0.65)
+                action[7] = np.float32(self._normalized_template(wrong_index))
+            elif macro_name in {"over_disturbance", "cloth_lift_high"}:
+                self._set_mode_bits(action, passive_mode)
+                action[0] = np.float32(1.0)
+                action[8] = np.float32(1.0 if macro_name == "over_disturbance" else 0.2)
+            elif macro_name == "delayed_actor_entry":
+                self._set_mode_bits(action, preferred_mode)
+                action[0] = np.float32(0.10)
+                action[8] = np.float32(0.2 if step_idx < (chunk_horizon - 1) else 1.0)
+            elif macro_name in {"weak_corridor_insert", "bag_fabric_probe"}:
+                self._set_mode_bits(action, passive_mode)
+                action[0] = np.float32(0.02)
+                action[8] = np.float32(1.0)
+            else:
+                action = self.teacher_action()
+            chunk[step_idx] = np.clip(action, -1.0, 1.0)
+        return chunk
+
+    def baseline_action_chunk(self, baseline_name: str, chunk_horizon: int = 8) -> np.ndarray:
+        if baseline_name == "teacher":
+            chunk, _ = self.teacher_chunk_and_rollout(chunk_horizon=chunk_horizon, rollout_horizon=self.rollout_horizon)
+            return chunk
+        if baseline_name == "reveal_only":
+            return self.macro_action_chunk("widen_gap" if self.proxy_name == FOLIAGE_PROXY.name else ("widen_mouth" if self.proxy_name == BAG_PROXY.name else "lift_edge"), chunk_horizon=chunk_horizon)
+        if baseline_name == "retrieve_only":
+            return self.macro_action_chunk("premature_retrieve", chunk_horizon=chunk_horizon)
+        if baseline_name == "no_hold":
+            return self.macro_action_chunk("reveal_with_release", chunk_horizon=chunk_horizon)
+        if baseline_name == "random":
+            return self.rng.uniform(-1.0, 1.0, size=(chunk_horizon, 14)).astype(np.float32)
+        raise KeyError(f"Unknown baseline chunk: {baseline_name}")
+
     def teacher_chunk_and_rollout(
         self,
         chunk_horizon: int = 8,
@@ -486,6 +663,7 @@ class ProceduralRevealEnv:
         rollout_reocclusion = []
         rollout_occluder_contact = []
         rollout_grasp_affordance = []
+        rollout_phase = []
         for step in range(chunk_horizon):
             action = self.teacher_action()
             action_chunk.append(action)
@@ -502,6 +680,7 @@ class ProceduralRevealEnv:
                 rollout_reocclusion.append(privileged_state["reocclusion_target"])
                 rollout_occluder_contact.append(privileged_state["occluder_contact_map"])
                 rollout_grasp_affordance.append(privileged_state["grasp_affordance_map"])
+                rollout_phase.append(int(privileged_state["phase_label"]))
             if terminated or truncated:
                 break
         while len(action_chunk) < chunk_horizon:
@@ -519,6 +698,7 @@ class ProceduralRevealEnv:
             rollout_reocclusion.append(current["reocclusion_target"])
             rollout_occluder_contact.append(current["occluder_contact_map"])
             rollout_grasp_affordance.append(current["grasp_affordance_map"])
+            rollout_phase.append(int(current["phase_label"]))
         self.restore_state(snapshot)
         return np.stack(action_chunk, axis=0).astype(np.float32), {
             "rollout_support_mode": np.asarray(rollout_support_mode, dtype=np.int64),
@@ -532,6 +712,7 @@ class ProceduralRevealEnv:
             "rollout_reocclusion_target": np.asarray(rollout_reocclusion, dtype=np.float32),
             "rollout_occluder_contact_map": np.asarray(rollout_occluder_contact, dtype=np.float32),
             "rollout_grasp_affordance_map": np.asarray(rollout_grasp_affordance, dtype=np.float32),
+            "rollout_phase": np.asarray(rollout_phase, dtype=np.int64),
         }
 
     def evaluate_action_chunk(
@@ -552,8 +733,12 @@ class ProceduralRevealEnv:
         rollout_reocclusion: list[float] = []
         rollout_occluder_contact: list[np.ndarray] = []
         rollout_grasp_affordance: list[np.ndarray] = []
+        rollout_phase: list[int] = []
         corridor_open_trace = [float(self.get_privileged_state()["corridor_feasible"][self._current_support_mode()].any())]
         visibility_trace = [float(self.get_privileged_state()["visibility"])]
+        disturbance_trace = [float(self.get_privileged_state()["disturbance_cost"])]
+        support_trace = [float(self.get_privileged_state()["support_stability"])]
+        opening_trace = [float(self.opening)]
         terminated = False
         truncated = False
         privileged_state = self.get_privileged_state()
@@ -571,8 +756,12 @@ class ProceduralRevealEnv:
                 rollout_reocclusion.append(float(privileged_state["reocclusion_target"]))
                 rollout_occluder_contact.append(privileged_state["occluder_contact_map"].astype(np.float32))
                 rollout_grasp_affordance.append(privileged_state["grasp_affordance_map"].astype(np.float32))
+                rollout_phase.append(int(privileged_state["phase_label"]))
             corridor_open_trace.append(float(privileged_state["corridor_feasible"][privileged_state["support_mode"]].any()))
             visibility_trace.append(float(privileged_state["visibility"]))
+            disturbance_trace.append(float(privileged_state["disturbance_cost"]))
+            support_trace.append(float(privileged_state["support_stability"]))
+            opening_trace.append(float(self.opening))
             if terminated or truncated:
                 break
         while len(rollout_support_mode) < rollout_horizon:
@@ -588,11 +777,17 @@ class ProceduralRevealEnv:
             rollout_reocclusion.append(float(current["reocclusion_target"]))
             rollout_occluder_contact.append(current["occluder_contact_map"].astype(np.float32))
             rollout_grasp_affordance.append(current["grasp_affordance_map"].astype(np.float32))
+            rollout_phase.append(int(current["phase_label"]))
         final_state = self.get_privileged_state()
+        corridor_curve = np.asarray(corridor_open_trace, dtype=np.float32)
+        visibility_curve = np.asarray(visibility_trace, dtype=np.float32)
+        disturbance_curve = np.asarray(disturbance_trace, dtype=np.float32)
+        support_curve = np.asarray(support_trace, dtype=np.float32)
+        opening_curve = np.asarray(opening_trace, dtype=np.float32)
         reocclusion = float(
             np.logical_and(
-                np.asarray(corridor_open_trace[:-1]) > 0.5,
-                np.asarray(corridor_open_trace[1:]) <= 0.5,
+                corridor_curve[:-1] > 0.5,
+                corridor_curve[1:] <= 0.5,
             ).mean()
         ) if len(corridor_open_trace) > 1 else 0.0
         result: dict[str, np.ndarray | float] = {
@@ -607,11 +802,29 @@ class ProceduralRevealEnv:
             "rollout_reocclusion_target": np.asarray(rollout_reocclusion, dtype=np.float32),
             "rollout_occluder_contact_map": np.asarray(rollout_occluder_contact, dtype=np.float32),
             "rollout_grasp_affordance_map": np.asarray(rollout_grasp_affordance, dtype=np.float32),
+            "rollout_phase": np.asarray(rollout_phase, dtype=np.int64),
             "retrieval_success": float(final_state["retrieval_success"]),
             "final_disturbance_cost": float(final_state["disturbance_cost"]),
             "reocclusion_rate": reocclusion,
-            "visibility_integral": float(np.sum(np.asarray(visibility_trace, dtype=np.float32))),
+            "visibility_integral": float(np.sum(visibility_curve)),
+            "actor_feasibility_auc": float(corridor_curve.mean()),
+            "reveal_achieved": float(visibility_curve.max() >= self.dynamics.retrieve_visibility_threshold),
+            "hold_persistence": float(corridor_curve.mean()),
+            "support_stability_auc": float(support_curve.mean()),
+            "disturbance_auc": float(disturbance_curve.mean()),
+            "opening_peak": float(opening_curve.max()),
         }
+        if self.proxy_name == FOLIAGE_PROXY.name:
+            result["candidate_gap_width"] = float(final_state.get("gap_width", opening_curve.max()))
+            result["candidate_damage_proxy"] = float(final_state.get("damage_proxy", final_state["disturbance_cost"]))
+        elif self.proxy_name == BAG_PROXY.name:
+            result["candidate_mouth_aperture"] = float(final_state.get("mouth_aperture", opening_curve.max()))
+            result["candidate_hold_quality"] = float(final_state.get("hold_quality", support_curve.mean()))
+            result["candidate_rim_slip_risk"] = float(final_state.get("rim_slip_risk", reocclusion))
+        elif self.proxy_name == CLOTH_PROXY.name:
+            result["candidate_fold_preservation"] = float(final_state.get("fold_preservation", 1.0 - final_state["disturbance_cost"]))
+            result["candidate_layer_separation_quality"] = float(final_state.get("layer_separation_quality", opening_curve.max()))
+            result["candidate_lift_too_much_risk"] = float(final_state.get("lift_too_much_risk", max(0.0, opening_curve.max() - self.dynamics.desired_opening)))
         self.restore_state(snapshot)
         return result
 
@@ -625,16 +838,70 @@ class ProceduralRevealEnv:
         teacher_chunk = np.asarray(teacher_chunk, dtype=np.float32)
         candidates = [teacher_chunk.astype(np.float32)]
         outcomes = [self.evaluate_action_chunk(teacher_chunk, rollout_horizon=rollout_horizon)]
-        for candidate_idx in range(1, num_candidates):
-            candidate = teacher_chunk.copy()
-            revealer_noise = self.rng.normal(loc=0.0, scale=0.20 + 0.03 * candidate_idx, size=candidate[:, :7].shape)
-            actor_noise = self.rng.normal(loc=0.0, scale=0.18 + 0.04 * candidate_idx, size=candidate[:, 7:].shape)
-            candidate[:, :7] = np.clip(candidate[:, :7] + revealer_noise.astype(np.float32), -1.0, 1.0)
-            candidate[:, 7:] = np.clip(candidate[:, 7:] + actor_noise.astype(np.float32), -1.0, 1.0)
+        candidate_macro_ids = [0]
+        candidate_is_hard_negative = [0.0]
+        candidate_macro_names = ["teacher"]
+        candidate_negative_families = ["teacher"]
+        if self.proxy_name == FOLIAGE_PROXY.name:
+            semantic_specs = [
+                ("pin_canopy", "positive"),
+                ("maintain_gap", "positive"),
+                ("premature_retrieve", "premature_retrieve"),
+                ("reveal_with_release", "reveal_with_release"),
+                ("wrong_side_reveal", "wrong_side_reveal"),
+                ("foliage_immediate_reocclusion", "immediate_reocclusion"),
+                ("over_disturbance", "over_disturbance"),
+                ("weak_corridor_insert", "weak_corridor_insert"),
+                ("insert_actor", "positive"),
+                ("retrieve", "positive"),
+            ]
+        elif self.proxy_name == BAG_PROXY.name:
+            semantic_specs = [
+                ("widen_mouth", "positive"),
+                ("maintain_mouth", "positive"),
+                ("premature_retrieve", "premature_retrieve"),
+                ("reveal_with_release", "reveal_with_release"),
+                ("wrong_edge_reveal", "wrong_side_reveal"),
+                ("pin_left_rim", "one_rim_slip"),
+                ("bag_fabric_probe", "fabric_probe"),
+                ("weak_corridor_insert", "weak_corridor_insert"),
+                ("insert_actor", "positive"),
+                ("retrieve", "positive"),
+            ]
+        else:
+            semantic_specs = [
+                ("lift_edge", "positive"),
+                ("stabilize_fold", "positive"),
+                ("premature_retrieve", "premature_retrieve"),
+                ("reveal_with_release", "reveal_with_release"),
+                ("cloth_lift_high", "lift_too_high"),
+                ("wrong_layer_reveal", "wrong_layer_reveal"),
+                ("delayed_actor_entry", "delayed_actor_entry"),
+                ("weak_corridor_insert", "weak_corridor_insert"),
+                ("insert_actor", "positive"),
+                ("retrieve", "positive"),
+            ]
+
+        for spec_idx, (macro_name, family_name) in enumerate(semantic_specs[: max(0, num_candidates - 1)], start=1):
+            candidate = self.macro_action_chunk(macro_name, chunk_horizon=teacher_chunk.shape[0])
             candidates.append(candidate.astype(np.float32))
             outcomes.append(self.evaluate_action_chunk(candidate, rollout_horizon=rollout_horizon))
+            candidate_macro_ids.append(spec_idx)
+            candidate_macro_names.append(macro_name)
+            candidate_negative_families.append(family_name)
+            candidate_is_hard_negative.append(0.0 if family_name == "positive" else 1.0)
+
+        while len(candidates) < num_candidates:
+            random_chunk = self.rng.uniform(-1.0, 1.0, size=teacher_chunk.shape).astype(np.float32)
+            candidates.append(random_chunk)
+            outcomes.append(self.evaluate_action_chunk(random_chunk, rollout_horizon=rollout_horizon))
+            candidate_macro_ids.append(len(candidate_macro_ids))
+            candidate_macro_names.append("random")
+            candidate_negative_families.append("random")
+            candidate_is_hard_negative.append(1.0)
         stacked_outcomes = {
             "candidate_rollout_support_mode": np.stack([item["rollout_support_mode"] for item in outcomes], axis=0).astype(np.int64),
+            "candidate_rollout_phase": np.stack([item["rollout_phase"] for item in outcomes], axis=0).astype(np.int64),
             "candidate_rollout_corridor_feasible": np.stack(
                 [item["rollout_corridor_feasible"] for item in outcomes], axis=0
             ).astype(np.float32),
@@ -671,6 +938,13 @@ class ProceduralRevealEnv:
             ),
             "candidate_reocclusion_rate": np.asarray([item["reocclusion_rate"] for item in outcomes], dtype=np.float32),
             "candidate_visibility_integral": np.asarray([item["visibility_integral"] for item in outcomes], dtype=np.float32),
+            "candidate_actor_feasibility_auc": np.asarray([item["actor_feasibility_auc"] for item in outcomes], dtype=np.float32),
+            "candidate_reveal_achieved": np.asarray([item["reveal_achieved"] for item in outcomes], dtype=np.float32),
+            "candidate_hold_persistence": np.asarray([item["hold_persistence"] for item in outcomes], dtype=np.float32),
+            "candidate_support_stability_auc": np.asarray([item["support_stability_auc"] for item in outcomes], dtype=np.float32),
+            "candidate_disturbance_auc": np.asarray([item["disturbance_auc"] for item in outcomes], dtype=np.float32),
+            "candidate_macro_ids": np.asarray(candidate_macro_ids, dtype=np.int64),
+            "candidate_is_hard_negative": np.asarray(candidate_is_hard_negative, dtype=np.float32),
         }
         stacked_outcomes["candidate_risk"] = np.clip(
             stacked_outcomes["candidate_final_disturbance_cost"] + stacked_outcomes["candidate_reocclusion_rate"],
@@ -680,6 +954,19 @@ class ProceduralRevealEnv:
         stacked_outcomes["candidate_utility"] = (
             stacked_outcomes["candidate_retrieval_success"] - stacked_outcomes["candidate_risk"]
         ).astype(np.float32)
+        stacked_outcomes["candidate_macro_names"] = candidate_macro_names
+        stacked_outcomes["candidate_negative_families"] = candidate_negative_families
+        if self.proxy_name == FOLIAGE_PROXY.name:
+            stacked_outcomes["candidate_gap_width"] = np.asarray([item["candidate_gap_width"] for item in outcomes], dtype=np.float32)
+            stacked_outcomes["candidate_damage_proxy"] = np.asarray([item["candidate_damage_proxy"] for item in outcomes], dtype=np.float32)
+        elif self.proxy_name == BAG_PROXY.name:
+            stacked_outcomes["candidate_mouth_aperture"] = np.asarray([item["candidate_mouth_aperture"] for item in outcomes], dtype=np.float32)
+            stacked_outcomes["candidate_hold_quality"] = np.asarray([item["candidate_hold_quality"] for item in outcomes], dtype=np.float32)
+            stacked_outcomes["candidate_rim_slip_risk"] = np.asarray([item["candidate_rim_slip_risk"] for item in outcomes], dtype=np.float32)
+        elif self.proxy_name == CLOTH_PROXY.name:
+            stacked_outcomes["candidate_fold_preservation"] = np.asarray([item["candidate_fold_preservation"] for item in outcomes], dtype=np.float32)
+            stacked_outcomes["candidate_layer_separation_quality"] = np.asarray([item["candidate_layer_separation_quality"] for item in outcomes], dtype=np.float32)
+            stacked_outcomes["candidate_lift_too_much_risk"] = np.asarray([item["candidate_lift_too_much_risk"] for item in outcomes], dtype=np.float32)
         return np.stack(candidates, axis=0).astype(np.float32), stacked_outcomes
 
     def step(self, action: np.ndarray) -> tuple[dict[str, Any], float, bool, bool, dict[str, Any]]:

code/reveal_vla_bimanual/train/configs/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact.yaml

@@ -0,0 +1,150 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17
+output_dir: /workspace/outputs/r3d_handoff
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies: [foliage_proxy, bag_proxy, cloth_proxy]
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 3
+  batch_size: 4
+  num_workers: 24
+  lr: 0.0001
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: compact_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 1.0
+  phase: 0.05
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.12
+  persistence: 0.06
+  disturbance: 0.06
+  world_model: 0.2
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.06
+  support_stability: 0.06
+  reocclusion: 0.06
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.2
+  planner_risk: 0.08
+  planner_ranking: 0.2
+  proposal_reconstruction: 0.08
+  proposal_success: 0.12
+  proposal_ranking: 0.15
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.05

code/reveal_vla_bimanual/train/configs/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase.yaml

@@ -0,0 +1,73 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17
+output_dir: /workspace/outputs/r3d_handoff_phase
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies: [foliage_proxy, bag_proxy, cloth_proxy]
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state_phase
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 3
+  batch_size: 4
+  num_workers: 24
+  lr: 0.0001
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone: {model_name: openai/clip-vit-base-patch32, hidden_dim: 512, max_text_tokens: 32, freeze_backbone: true, gradient_checkpointing: false, use_dummy_backbone: false}
+  fusion: {hidden_dim: 512, num_cameras: 3, num_layers: 4, num_heads: 8, ff_dim: 2048, dropout: 0.1, proprio_dim: 32, proprio_tokens: 1}
+  memory: {hidden_dim: 512, action_dim: 14, history_steps: 6, scene_history_steps: 3, belief_history_steps: 8, num_layers: 2, dropout: 0.1, memory_bank_size: 4, scene_bank_size: 2, belief_bank_size: 2, num_heads: 8, max_history_steps: 8}
+  decoder: {hidden_dim: 512, num_heads: 8, num_layers: 4, ff_dim: 2048, dropout: 0.1, chunk_size: 8, action_dim: 14, arm_action_dim: 7, num_candidates: 8, num_phases: 5, num_arm_roles: 4, num_proposal_modes: 7, planner_top_k: 4}
+  reveal_head: {hidden_dim: 512, num_support_modes: 3, num_approach_templates: 32, rollout_horizon: 5, belief_map_size: 32, field_size: 16, num_heads: 8, predict_belief_map: true, num_phases: 5, num_arm_roles: 4, num_interaction_tokens: 8, num_tasks: 4}
+  world_model: {hidden_dim: 512, action_dim: 14, num_support_modes: 3, num_approach_templates: 32, rollout_horizon: 5, field_size: 16, num_heads: 8, num_phases: 5, num_arm_roles: 4, num_interaction_tokens: 8, belief_map_size: 32, predict_belief_map: true, scene_bank_size: 2, belief_bank_size: 2, rollout_mode: compact_rollout, num_tasks: 4}
+  planner: {hidden_dim: 512, num_candidates: 8, action_dim: 14, num_support_modes: 3, utility_margin: 0.1, num_heads: 8, num_layers: 2, num_phases: 5, num_arm_roles: 4, top_k: 4}
+loss_weights:
+  action: 1.0
+  phase: 0.08
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.12
+  persistence: 0.06
+  disturbance: 0.06
+  world_model: 0.2
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.06
+  support_stability: 0.06
+  reocclusion: 0.06
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.2
+  planner_risk: 0.08
+  planner_ranking: 0.2
+  proposal_reconstruction: 0.08
+  proposal_success: 0.12
+  proposal_ranking: 0.15
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.05

code/reveal_vla_bimanual/train/configs/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial.yaml

@@ -0,0 +1,150 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17
+output_dir: /workspace/outputs/r3d_handoff
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies: [foliage_proxy, bag_proxy, cloth_proxy]
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 5
+  batch_size: 4
+  num_workers: 24
+  lr: 0.00015
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone:
+    model_name: openai/clip-vit-base-patch32
+    hidden_dim: 512
+    max_text_tokens: 32
+    freeze_backbone: true
+    gradient_checkpointing: false
+    use_dummy_backbone: false
+  fusion:
+    hidden_dim: 512
+    num_cameras: 3
+    num_layers: 4
+    num_heads: 8
+    ff_dim: 2048
+    dropout: 0.1
+    proprio_dim: 32
+    proprio_tokens: 1
+  memory:
+    hidden_dim: 512
+    action_dim: 14
+    history_steps: 6
+    scene_history_steps: 3
+    belief_history_steps: 8
+    num_layers: 2
+    dropout: 0.1
+    memory_bank_size: 4
+    scene_bank_size: 2
+    belief_bank_size: 2
+    num_heads: 8
+    max_history_steps: 8
+  decoder:
+    hidden_dim: 512
+    num_heads: 8
+    num_layers: 4
+    ff_dim: 2048
+    dropout: 0.1
+    chunk_size: 8
+    action_dim: 14
+    arm_action_dim: 7
+    num_candidates: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_proposal_modes: 7
+    planner_top_k: 4
+  reveal_head:
+    hidden_dim: 512
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    belief_map_size: 32
+    field_size: 16
+    num_heads: 8
+    predict_belief_map: true
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    num_tasks: 4
+  world_model:
+    hidden_dim: 512
+    action_dim: 14
+    num_support_modes: 3
+    num_approach_templates: 32
+    rollout_horizon: 5
+    field_size: 16
+    num_heads: 8
+    num_phases: 5
+    num_arm_roles: 4
+    num_interaction_tokens: 8
+    belief_map_size: 32
+    predict_belief_map: true
+    scene_bank_size: 2
+    belief_bank_size: 2
+    rollout_mode: spatial_rollout
+    num_tasks: 4
+  planner:
+    hidden_dim: 512
+    num_candidates: 8
+    action_dim: 14
+    num_support_modes: 3
+    utility_margin: 0.1
+    num_heads: 8
+    num_layers: 2
+    num_phases: 5
+    num_arm_roles: 4
+    top_k: 4
+loss_weights:
+  action: 0.6
+  phase: 0.05
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.15
+  persistence: 0.08
+  disturbance: 0.08
+  world_model: 0.35
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.08
+  support_stability: 0.08
+  reocclusion: 0.08
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.25
+  planner_risk: 0.1
+  planner_ranking: 0.25
+  proposal_reconstruction: 0.05
+  proposal_success: 0.2
+  proposal_ranking: 0.25
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.1

code/reveal_vla_bimanual/train/configs/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase.yaml

@@ -0,0 +1,73 @@
+experiment_name: proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17
+output_dir: /workspace/outputs/r3d_handoff_phase
+device: cuda
+seed: 17
+init_checkpoint: /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt
+init_strict: false
+data:
+  proxies: [foliage_proxy, bag_proxy, cloth_proxy]
+  resolution: 224
+  dataset_version: reveal_proxy_v6_rgbd_elastic_state_phase
+  train_episodes_per_proxy: 48
+  val_episodes_per_proxy: 16
+  train_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt
+  val_dataset_path: /workspace/VLAarchtests/artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt
+  rebuild_dataset: false
+  chunk_horizon: 8
+  rollout_horizon: 5
+  history_steps: 6
+  planner_candidates: 8
+  seed: 17
+optim:
+  epochs: 4
+  batch_size: 4
+  num_workers: 24
+  lr: 0.00015
+  weight_decay: 0.0001
+trainer:
+  policy_type: elastic_reveal
+  use_bf16: true
+  grad_clip_norm: 1.0
+  freeze_backbone: true
+  gradient_checkpointing: false
+  plan_during_train: true
+  plan_during_eval: true
+  support_mode_conditioning: true
+  planner_mode: trainable
+  use_depth: true
+  use_world_model: true
+  use_role_tokens: true
+  compute_equivariance_probe: false
+policy:
+  backbone: {model_name: openai/clip-vit-base-patch32, hidden_dim: 512, max_text_tokens: 32, freeze_backbone: true, gradient_checkpointing: false, use_dummy_backbone: false}
+  fusion: {hidden_dim: 512, num_cameras: 3, num_layers: 4, num_heads: 8, ff_dim: 2048, dropout: 0.1, proprio_dim: 32, proprio_tokens: 1}
+  memory: {hidden_dim: 512, action_dim: 14, history_steps: 6, scene_history_steps: 3, belief_history_steps: 8, num_layers: 2, dropout: 0.1, memory_bank_size: 4, scene_bank_size: 2, belief_bank_size: 2, num_heads: 8, max_history_steps: 8}
+  decoder: {hidden_dim: 512, num_heads: 8, num_layers: 4, ff_dim: 2048, dropout: 0.1, chunk_size: 8, action_dim: 14, arm_action_dim: 7, num_candidates: 8, num_phases: 5, num_arm_roles: 4, num_proposal_modes: 7, planner_top_k: 4}
+  reveal_head: {hidden_dim: 512, num_support_modes: 3, num_approach_templates: 32, rollout_horizon: 5, belief_map_size: 32, field_size: 16, num_heads: 8, predict_belief_map: true, num_phases: 5, num_arm_roles: 4, num_interaction_tokens: 8, num_tasks: 4}
+  world_model: {hidden_dim: 512, action_dim: 14, num_support_modes: 3, num_approach_templates: 32, rollout_horizon: 5, field_size: 16, num_heads: 8, num_phases: 5, num_arm_roles: 4, num_interaction_tokens: 8, belief_map_size: 32, predict_belief_map: true, scene_bank_size: 2, belief_bank_size: 2, rollout_mode: spatial_rollout, num_tasks: 4}
+  planner: {hidden_dim: 512, num_candidates: 8, action_dim: 14, num_support_modes: 3, utility_margin: 0.1, num_heads: 8, num_layers: 2, num_phases: 5, num_arm_roles: 4, top_k: 4}
+loss_weights:
+  action: 0.6
+  phase: 0.08
+  arm_role: 0.1
+  support_mode: 0.1
+  corridor: 0.15
+  persistence: 0.08
+  disturbance: 0.08
+  world_model: 0.35
+  belief: 0.05
+  visibility: 0.05
+  clearance: 0.08
+  support_stability: 0.08
+  reocclusion: 0.08
+  occluder_contact: 0.05
+  grasp_affordance: 0.05
+  planner_success: 0.25
+  planner_risk: 0.1
+  planner_ranking: 0.25
+  proposal_reconstruction: 0.05
+  proposal_success: 0.2
+  proposal_ranking: 0.25
+  proposal_diversity: 0.05
+  role_swap_consistency: 0.02
+  task_metrics: 0.1

code/reveal_vla_bimanual/train/losses.py

@@ -32,6 +32,7 @@ class LossWeights:
     proposal_ranking: float = 0.05
     proposal_diversity: float = 0.05
     role_swap_consistency: float = 0.05
+    task_metrics: float = 0.05
 
 
 def chunk_bc_loss(pred_actions: Tensor, target_actions: Tensor, mask: Tensor | None = None) -> Tensor:
@@ -113,12 +114,15 @@ def _resize_like(target: Tensor, prediction: Tensor) -> Tensor:
 def reveal_state_loss(pred: dict[str, Tensor], target: dict[str, Tensor], weights: LossWeights) -> dict[str, Tensor]:
     losses = {}
     if "phase_logits" in pred:
-        action_chunk = target.get("action_chunk")
-        if action_chunk is not None:
-            phase_target = infer_phase_targets_from_actions(action_chunk[:, 0])
+        if "phase" in target:
+            phase_target = target["phase"].long()
         else:
-            phase_map = torch.as_tensor([2, 3, 0], device=target["support_mode"].device, dtype=torch.long)
-            phase_target = phase_map[target["support_mode"].long()]
+            action_chunk = target.get("action_chunk")
+            if action_chunk is not None:
+                phase_target = infer_phase_targets_from_actions(action_chunk[:, 0])
+            else:
+                phase_map = torch.as_tensor([2, 3, 0], device=target["support_mode"].device, dtype=torch.long)
+                phase_target = phase_map[target["support_mode"].long()]
         losses["phase"] = F.cross_entropy(pred["phase_logits"], phase_target)
     else:
         losses["phase"] = pred["support_mode_logits"].new_tensor(0.0)
@@ -190,6 +194,33 @@ def reveal_state_loss(pred: dict[str, Tensor], target: dict[str, Tensor], weight
         losses["uncertainty"] = pred["persistence_uncertainty"].mean()
     else:
         losses["uncertainty"] = pred["support_mode_logits"].new_tensor(0.0)
+    task_metric_pairs = (
+        "opening_quality",
+        "actor_feasibility_score",
+        "gap_width",
+        "damage_proxy",
+        "release_collapse_rate",
+        "target_visibility_confidence",
+        "mouth_aperture",
+        "hold_quality",
+        "rim_slip_risk",
+        "insertable_actor_corridor",
+        "layer_separation_quality",
+        "fold_preservation",
+        "insertion_corridor",
+        "top_layer_stability",
+        "lift_too_much_risk",
+    )
+    task_losses = [
+        F.mse_loss(pred[key].float(), target[key].float())
+        for key in task_metric_pairs
+        if key in pred and key in target
+    ]
+    losses["task_metrics"] = (
+        torch.stack(task_losses).mean()
+        if task_losses
+        else pred["support_mode_logits"].new_tensor(0.0)
+    )
     return losses
 
 
@@ -221,6 +252,8 @@ def world_model_rollout_consistency_loss(pred_rollout: dict[str, Tensor], target
         "disturbance_cost": _expand_target(target_rollout["disturbance_cost"][..., :horizon]),
         "action_chunk": _expand_target(target_rollout["action_chunk"][..., :horizon, :]),
     }
+    if "phase" in target_rollout:
+        target_rollout["phase"] = _expand_target(target_rollout["phase"][..., :horizon])
     loss = (
         F.cross_entropy(
             pred_rollout["support_mode_logits"].reshape(-1, pred_rollout["support_mode_logits"].shape[-1]),
@@ -234,7 +267,9 @@ def world_model_rollout_consistency_loss(pred_rollout: dict[str, Tensor], target
         + F.mse_loss(pred_rollout["disturbance_cost"], target_rollout["disturbance_cost"].float())
     )
     if "phase_logits" in pred_rollout:
-        phase_target = infer_phase_targets_from_actions(target_rollout["action_chunk"])
+        phase_target = target_rollout.get("phase")
+        if phase_target is None:
+            phase_target = infer_phase_targets_from_actions(target_rollout["action_chunk"])
         loss = loss + 0.5 * F.cross_entropy(
             pred_rollout["phase_logits"].reshape(-1, pred_rollout["phase_logits"].shape[-1]),
             phase_target.reshape(-1),
@@ -300,6 +335,7 @@ def compute_total_loss(
             + weights.occluder_contact * reveal_losses["occluder_contact"]
             + weights.grasp_affordance * reveal_losses["grasp_affordance"]
             + weights.reocclusion * reveal_losses["reocclusion"]
+            + weights.task_metrics * reveal_losses["task_metrics"]
             + 0.01 * reveal_losses["uncertainty"]
         )
 
@@ -314,6 +350,8 @@ def compute_total_loss(
                 "disturbance_cost": batch["candidate_rollout_disturbance_cost"],
                 "action_chunk": batch["candidate_action_chunks"],
             }
+            if "candidate_rollout_phase" in batch:
+                rollout_target["phase"] = batch["candidate_rollout_phase"]
             for optional_key in (
                 "candidate_rollout_belief_map",
                 "candidate_rollout_visibility_map",
@@ -344,6 +382,8 @@ def compute_total_loss(
                 "disturbance_cost": batch["rollout_disturbance_cost"],
                 "action_chunk": batch["action_chunk"],
             }
+            if "rollout_phase" in batch:
+                rollout_target["phase"] = batch["rollout_phase"]
             for optional_key in (
                 "rollout_belief_map",
                 "rollout_visibility_map",

results/2026-03-25-runpod/README.md

@@ -0,0 +1,124 @@
+# 2026-03-25 Runpod Raw Index
+
+This directory contains the source handoff instructions, raw report files copied from `/workspace/reports`, and a raw index for the artifacts produced in the `2026-03-25 UTC` session.
+
+## Source Handoff
+
+- `instructions.md`
+
+## Test Suite
+
+- Command:
+  - `PYTHONPATH=/workspace/VLAarchtests_work/code/reveal_vla_bimanual python -m pytest -q /workspace/VLAarchtests_work/tests`
+- Result:
+  - `33 passed`
+
+## Generated Datasets
+
+| Path | Size (bytes) |
+| --- | ---: |
+| `artifacts/data/reveal_proxy/proxy_train_clip224_v6_rgbd_stage3_phase.pt` | 583377508 |
+| `artifacts/data/reveal_proxy/proxy_val_clip224_v6_rgbd_stage3_phase.pt` | 200844508 |
+
+## Generated Checkpoints And Training Summaries
+
+| Directory | final_train_total | final_val_total | train_time_sec | peak_gpu_memory_mb |
+| --- | ---: | ---: | ---: | ---: |
+| `artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_seed17/` | 0.382780 | 0.372276 | 108.922691 | 2451.385742 |
+| `artifacts/outputs/r3d_handoff/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_seed17/` | 0.518003 | 0.503869 | 163.313406 | 2924.821777 |
+| `artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_seed17/` | 0.385303 | 0.378928 | 128.965583 | 2450.287109 |
+| `artifacts/outputs/r3d_handoff_phase/proxy_interaction_r3d_stage3_clip_rgbd_handoff_spatial_phase_seed17/` | 0.525366 | 0.507625 | 154.841441 | 2926.074707 |
+
+## Proxy Result Files
+
+### Serious Comparisons
+
+| File | Reference mean success | Compared mean success | Compared foliage | Compared bag | Compared cloth |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| `reports/reveal_handoff_compare_serious/reveal_benchmark.json` | 0.583333 | 0.216667 | 0.330000 | 0.150000 | 0.170000 |
+| `reports/reveal_handoff_compare_serious_compact/reveal_benchmark.json` | 0.583333 | 0.520000 | 0.660000 | 0.320000 | 0.580000 |
+| `reports/reveal_phase_compare_serious_compact/reveal_benchmark.json` | 0.583333 | 0.513333 | 0.570000 | 0.420000 | 0.550000 |
+| `reports/reveal_phase_compare_serious_spatial_compactwm/reveal_benchmark.json` | 0.583333 | 0.493333 | 0.640000 | 0.330000 | 0.510000 |
+
+### Compact-Phase Ablation Matrix
+
+| Ablation | mean_success | visibility_integral | reocclusion_rate | disturbance_cost |
+| --- | ---: | ---: | ---: | ---: |
+| `full_model` | 0.513333 | 39.978670 | 0.000000 | 0.343669 |
+| `no_geometry` | 0.513333 | 39.983892 | 0.000000 | 0.343637 |
+| `no_spatial_memory` | 0.496667 | 37.758093 | 0.002417 | 0.417673 |
+| `compact_world_model` | 0.513333 | 39.978670 | 0.000000 | 0.343669 |
+| `no_planner` | 0.433333 | 20.634101 | 0.019708 | 0.185775 |
+| `gaussian_candidates_only` | 0.466667 | 16.719086 | 0.029561 | 0.477573 |
+| `no_task_head` | 0.513333 | 38.128876 | 0.000000 | 0.344494 |
+| `no_support_mode_conditioning` | 0.513333 | 39.978670 | 0.000000 | 0.343669 |
+
+Files:
+
+- `reports/reveal_phase_ablations_compact/ablations.json`
+- `reports/reveal_phase_ablations_compact/ablations.md`
+- `reports/reveal_phase_ablations_compact/ablations.partial.json`
+- `reports/reveal_phase_ablations_spatial/ablations.partial.json`
+
+### Teacher Audit
+
+| Proxy | Baseline | teacher_success | baseline_success | success_delta | teacher_utility | baseline_utility | utility_delta |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| `foliage_proxy` | `reveal_only` | 1.000000 | 0.000000 | 1.000000 | 1.198432 | 0.140528 | 1.057904 |
+| `foliage_proxy` | `retrieve_only` | 1.000000 | 0.000000 | 1.000000 | 1.198432 | -0.099185 | 1.297617 |
+| `foliage_proxy` | `no_hold` | 1.000000 | 0.000000 | 1.000000 | 1.198432 | 0.084153 | 1.114280 |
+| `foliage_proxy` | `random` | 1.000000 | 0.000000 | 1.000000 | 1.198432 | -0.138381 | 1.336814 |
+| `bag_proxy` | `reveal_only` | 1.000000 | 0.000000 | 1.000000 | 1.196591 | 0.167617 | 1.028974 |
+| `bag_proxy` | `retrieve_only` | 1.000000 | 0.000000 | 1.000000 | 1.196591 | -0.161481 | 1.358072 |
+| `bag_proxy` | `no_hold` | 1.000000 | 0.000000 | 1.000000 | 1.196591 | 0.078455 | 1.118136 |
+| `bag_proxy` | `random` | 1.000000 | 0.000000 | 1.000000 | 1.196591 | -0.181732 | 1.378323 |
+| `cloth_proxy` | `reveal_only` | 1.000000 | 0.000000 | 1.000000 | 1.276440 | 0.347192 | 0.929249 |
+| `cloth_proxy` | `retrieve_only` | 1.000000 | 0.000000 | 1.000000 | 1.276440 | 0.001142 | 1.275299 |
+| `cloth_proxy` | `no_hold` | 1.000000 | 0.000000 | 1.000000 | 1.276440 | 0.507900 | 0.768540 |
+| `cloth_proxy` | `random` | 1.000000 | 0.010000 | 0.990000 | 1.276440 | 0.166110 | 1.110330 |
+
+Files:
+
+- `reports/reveal_teacher_audit_serious/teacher_audit.json`
+- `reports/reveal_teacher_audit_serious/teacher_audit.md`
+
+### Additional Proxy Report Files
+
+- `reports/reveal_smoke_mod/reveal_benchmark.json`
+- `reports/reveal_smoke_nogeom/reveal_benchmark.json`
+- `reports/reveal_smoke_noplanner/reveal_benchmark.json`
+- `reports/reveal_handoff_compact_probe/reveal_benchmark.json`
+- `reports/reveal_handoff_compact_train_probe/reveal_benchmark.json`
+- `reports/reveal_phase_probe_compact/reveal_benchmark.json`
+- `reports/reveal_phase_probe_spatial/reveal_benchmark.json`
+- `reports/reveal_phase_probe_spatial_compactwm/reveal_benchmark.json`
+
+## RLBench Result Files
+
+### Full-Split PerAct2 Rollout Outputs
+
+| File | plan_requested | plan_applied | mean_success |
+| --- | --- | --- | ---: |
+| `reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/rollout_eval.json` | `true` | `true` | 0.000000 |
+| `reports/peract2_spatial_full_ep1/spatial_phase_seed17_noplan_split/rollout_eval.json` | `false` | `false` | 0.000000 |
+| `reports/peract2_spatial_full_ep1/spatial_phase_seed17_plan_split/rollout_eval.json` | `true` | `true` | 0.000000 |
+
+### Single-Task RLBench Debug Outputs
+
+- `reports/rlbench_debug_baseline_pushbox/rollout_eval.json`
+- `reports/rlbench_debug_compact_pushbox/rollout_eval.json`
+- `reports/rlbench_debug_spatial_pushbox_nogeom/rollout_eval.json`
+
+## Environment Recreation Files
+
+- `environment/README.md`
+- `environment/setup_same_machine.sh`
+- `environment/validate_same_machine.sh`
+- `environment/runtime_env_vars.sh`
+- `environment/hardware_snapshot.txt`
+- `environment/glxinfo_B.txt`
+- `environment/upstream_revisions.txt`
+- `environment/system_packages_same_machine.txt`
+- `environment/rlbench_env_export.yaml`
+- `environment/rlbench_env_explicit.txt`
+- `environment/rlbench_pip_freeze.txt`

results/2026-03-25-runpod/instructions.md

@@ -0,0 +1,717 @@
+# Developer handoff: structured bimanual reveal-and-retrieve under elastic occlusion
+
+Repo target: `lsnu/VLAarchtests` (current `main`, latest post-fix state). This handoff is written against the current `elastic_reveal` stack, not the older intermediate variants.
+
+## 1. Project introduction
+
+This project is a structured bimanual policy stack for reveal-and-retrieve tasks under partial observability and deformable or elastic occlusion. The eventual real-world targets are three Dobot X-trainer environments. The first is dense live foliage with hidden fake snails, where one arm must create and maintain a canopy gap while the other arm retrieves the target safely. The second is bag opening and retrieval, where one arm must open and hold the bag mouth while the other arm retrieves the target item. The third is suitcase or folded-cloth retrieval, where one arm must slightly lift and stabilize clothing layers while the other arm retrieves a hidden item without destroying the fold structure.
+
+The current repo already contains the right broad decomposition for this task family. It has a multi-view visual backbone, RGB-D support, an explicit reveal state head, observation memory, a compact world model, a coordinated bimanual action decoder, and a planner. The problem is not the structural idea. The problem is that several important pieces are only partially wired, too compact, or only validated on teacher-shaped proxy data. The current code is a good scaffold. It is not yet strong enough to justify “beats SOTA” claims on either public benchmarks or the three target task families.
+
+The current public evidence should be read narrowly. The most credible positive result in the repo is that RGB-D helps on the proxy benchmark. The planner, world model, and role-symmetry components are not yet validated strongly enough to claim they are the source of the gains. The RLBench / PerAct2 integration is also still mostly a launch and plumbing layer, not a mature benchmark suite.
+
+This handoff therefore has one purpose. Keep the structured reveal-and-retrieve idea, but harden the architecture and evaluation until there is a realistic chance of beating strong bimanual baselines on the three target environments.
+
+## 2. Current repo status (what exists, what is missing)
+
+The current core files are:
+
+`code/reveal_vla_bimanual/models/backbones.py`  
+`code/reveal_vla_bimanual/models/multiview_fusion.py`  
+`code/reveal_vla_bimanual/models/observation_memory.py`  
+`code/reveal_vla_bimanual/models/reveal_head.py`  
+`code/reveal_vla_bimanual/models/world_model.py`  
+`code/reveal_vla_bimanual/models/action_decoder.py`  
+`code/reveal_vla_bimanual/models/planner.py`  
+`code/reveal_vla_bimanual/models/policy.py`  
+`code/reveal_vla_bimanual/train/losses.py`  
+`code/reveal_vla_bimanual/sim_reveal/dataset.py`  
+`code/reveal_vla_bimanual/sim_reveal/procedural_envs.py`  
+`code/reveal_vla_bimanual/eval/run_reveal_benchmark.py`  
+`code/reveal_vla_bimanual/eval/run_rlbench_rollout_eval.py`  
+`code/reveal_vla_bimanual/eval/run_peract2_task_sweep.py`
+
+The current proxy benchmark already uses the correct three abstract task types (`foliage`, `bag`, `cloth`). That is good. The current dataset code also has explicit no-leak assertions, which is also good.
+
+The current weaknesses are specific and fixable.
+
+First, the geometry path is only partially wired. The backbone produces `depth_tokens`, `geometry_tokens`, and `camera_tokens`, but the policy only forwards RGB, depth, and camera tokens into fusion. The explicit `geometry_tokens` are dropped before fusion. In addition, camera geometry is incomplete. The current depth adapter encodes intrinsics and camera translation, but not an equally explicit camera rotation representation. For three-camera reveal tasks this is a real omission.
+
+Second, memory is too pooled and too global. The current memory path reduces scene history to pooled tokens before write decisions and bank updates. That is a novelty-gated summary memory. It is not a spatial occlusion memory. That is not enough for “hold the opening”, “the target is still probably behind this flap”, or “reveal progress will collapse if the revealer arm releases now”.
+
+Third, the world model is too compact. It is useful as a scaffold, but not as the state-transition core for elastic foliage, bag apertures, or layered cloth. It currently rolls a compact hidden state rather than a spatial field state. That makes it too weak for counterfactual planning over opening persistence, reocclusion, and safe actor insertion.
+
+Fourth, the planner is not trained on hard enough candidates. The current proxy data generation uses the teacher chunk and mostly Gaussian perturbations around it. That is enough to test ranking near a teacher, but not enough to teach the planner the actual failure modes that matter in these tasks (premature retrieval, releasing the opening, over-disturbing the scene, lifting the wrong cloth edge, etc.).
+
+Fifth, the state head is still too generic. It predicts a useful set of reveal-related fields, but it does not yet expose the right task-specific latent variables for foliage, bag, and folded cloth. Those tasks are not the same. They share the same reveal-and-retrieve pattern, but they do not share the same dominant failure modes.
+
+Sixth, the test suite is mostly contract-level. Those tests are useful, but they do not yet prove that the structured components work behaviorally. The RLBench side is similar. The launch smoke is only a plumbing check. The actual rollout evaluator exists, but it needs to become the main public benchmark path.
+
+## 3. The main design decision
+
+Do not collapse this into a generic monolithic VLA. That is not the likely win condition for these tasks.
+
+The highest-probability path is a stronger visual backbone plus an explicit structured reveal-and-retrieve stack. The reason is simple. Your target tasks are asymmetric, partially observable, persistence-sensitive, and reocclusion-sensitive. One arm often has to create and maintain a temporary affordance that only exists because of that arm’s continued state. Generic end-to-end BC can sometimes imitate the behavior, but these tasks strongly reward explicit representations of opening quality, hold persistence, target belief, reocclusion risk, and actor feasibility.
+
+The structured architecture should stay. It should just become spatial, task-aware, and evaluated honestly.
+
+## 4. Mandatory code changes
+
+### 4.1 Fix and strengthen the geometry path
+
+Files to change:
+
+`models/backbones.py`  
+`models/multiview_fusion.py`  
+`models/policy.py`  
+`tests/test_rgbd_forward_contract.py` (extend)  
+Add new tests: `tests/test_geometry_tokens_propagate.py`, `tests/test_camera_rotation_geometry.py`
+
+Exact changes:
+
+In `models/policy.py`, update the image encoding path so that `geometry_tokens` are passed from `backbone.encode_images(..., return_aux=True)` into the fusion module. Right now the policy forwards `rgb_tokens`, `depth_tokens`, and `camera_tokens`, but not `geometry_tokens`. This should be corrected first because it is an actual information-drop bug.
+
+In `models/multiview_fusion.py`, update the fusion interface to accept explicit `geometry_tokens`. The geometry attention path should fuse from a real concatenation or gated combination of `[depth_tokens, geometry_tokens, camera_tokens]`, rather than synthesizing “geometry” only from the surviving depth and camera paths. Keep the existing gated cross-attention pattern, but make the geometry path explicit and inspectable.
+
+In `models/backbones.py`, upgrade `DepthPatchAdapter` so that geometry features include camera orientation. Use a 6D rotation representation or a normalized quaternion plus translation. Also add per-patch viewing ray directions derived from intrinsics and camera pose. The three target environments all rely on view geometry and persistent multi-view correspondence. The current translation-only pose treatment is too weak.
+
+Add config flags that actually do something. The current `use_camera_geometry` style config needs to gate a real path, not just exist as a dormant option. Add separate switches for `use_depth_tokens`, `use_geometry_tokens`, and `use_camera_pose_tokens` so ablations are clean.
+
+Why this matters: the foliage and bag tasks are especially sensitive to camera geometry because small apparent gaps can be fake from one viewpoint and usable from another. The actor feasibility estimate should depend on geometry, not just appearance.
+
+### 4.2 Replace pooled novelty memory with spatial reveal memory
+
+Files to change:
+
+`models/observation_memory.py`  
+`models/policy.py`  
+`models/reveal_head.py`  
+Add new tests: `tests/test_spatial_memory_occlusion_persistence.py`, `tests/test_memory_slot_write_gating.py`, `tests/test_reocclusion_memory_regression.py`
+
+Exact changes:
+
+Keep the current memory modules as a fallback baseline, but add a new default path that stores low-resolution spatial memory instead of only pooled history summaries. The simplest realistic version is a two-branch memory:
+
+1. scene memory: a small bank of view-conditioned or canonicalized spatial tokens for persistent geometry and support structure;
+2. belief memory: a spatial target-belief / reveal-state memory that carries uncertainty explicitly.
+
+The memory does not need to be large. An 8×8 or 12×12 field token grid per view (or a shared canonical field) is enough. The key requirement is that the write gate becomes spatial or slot-wise, not global only. The model must be able to update “the mouth is open here” without overwriting “the target is probably still here”.
+
+Add explicit channels or latent heads for:
+- newly revealed regions
+- still-visible regions
+- reoccluded regions
+- persistent hold or opening quality
+- target belief uncertainty
+
+The world model and planner should consume this spatial memory directly. Do not average it away before planning.
+
+Why this matters: a reveal-and-retrieve policy that forgets where the useful opening was, or where the hidden object probably still is, will look competent in one-step imitation and fail in multi-step retrieval.
+
+### 4.3 Replace the compact world model with a spatial rollout model
+
+Files to change:
+
+`models/world_model.py`  
+`models/policy.py`  
+`train/losses.py`  
+Add new tests: `tests/test_world_model_null_rollout.py`, `tests/test_world_model_identity_rollout.py`, `tests/test_world_model_field_consistency.py`, `tests/test_world_model_task_adapter.py`
+
+Exact changes:
+
+Keep the current compact GRU world model only as an ablation. The default model should become a spatial latent rollout over field tokens or low-resolution maps. A realistic implementation is a ConvGRU or a token-wise recurrent transformer over a low-resolution field state. The world-model state should contain at least:
+
+- target belief field
+- visibility or reveal field
+- actor feasibility / corridor field
+- opening quality or hold quality field
+- persistence field
+- disturbance / damage risk field
+- reocclusion risk field
+- support stability field
+
+Add task conditioning directly into the world model. A learned task embedding (`foliage`, `bag`, `cloth`) should modulate the transition. The dynamics are not the same and should not be forced into one unstructured transition model.
+
+Retain explicit ablation modes inside `models/world_model.py`:
+- `identity_rollout`
+- `null_rollout`
+- `compact_rollout` (the current baseline)
+- `spatial_rollout` (new default)
+
+These ablations must be real and deterministic. The world-model ablation confusion in the current repo shows why this needs to be explicit and unit-tested.
+
+Why this matters: the planner can only beat a simple decoder if its counterfactual rollouts capture persistence and collapse. Without a spatial world model, the “maintain opening while actor advances” pattern will be under-modeled.
+
+### 4.4 Make the reveal head task-aware
+
+Files to change:
+
+`models/reveal_head.py`  
+`train/losses.py`  
+`sim_reveal/dataset.py`  
+`sim_reveal/procedural_envs.py`  
+Add new tests: `tests/test_task_conditioned_head_shapes.py`, `tests/test_task_metric_monotonicity.py`
+
+Exact changes:
+
+Add a task embedding to the reveal head. Keep the shared trunk, but use task-specific adapters or low-rank heads for the final outputs. The head should still produce common fields, but each task must also expose the state variables that actually matter.
+
+For foliage, add:
+- gap width or reveal corridor width
+- canopy strain / damage risk
+- occluder return tendency (reocclusion after release)
+- target visibility confidence under flexible occluders
+
+For bag, add:
+- mouth aperture width or area
+- rim endpoint or rim grasp quality
+- hold quality
+- rim slip risk
+- insertable actor corridor
+
+For cloth or suitcase, add:
+- layer separation quality
+- fold-preservation score
+- insertion corridor
+- top-layer stability
+- “lift too much” risk
+
+The current generic fields (`actor_feasibility_field`, `persistence_field`, `risk_field`, `uncertainty_field`, `reocclusion`) are useful, but they are not enough. The planner needs the task-specific variables because the right action for bag opening is not the right action for layered cloth.
+
+### 4.5 Replace Gaussian candidate noise with semantic macro candidates plus continuous refinement
+
+Files to change:
+
+`models/action_decoder.py`  
+`models/planner.py`  
+`sim_reveal/dataset.py`  
+`sim_reveal/procedural_envs.py`  
+Add new tests: `tests/test_candidate_macro_coverage.py`, `tests/test_planner_reocclusion_gating.py`, `tests/test_proposal_semantic_diversity.py`
+
+Exact changes:
+
+Keep the current proposal mechanism as a fallback. The default candidate set should become a set of semantic macro modes, each refined by continuous deltas.
+
+The candidate vocabulary should be task-aware.
+
+For foliage:
+- `sweep_left`
+- `sweep_right`
+- `pin_canopy`
+- `widen_gap`
+- `maintain_gap`
+- `insert_actor`
+- `retrieve`
+
+For bag:
+- `pin_left_rim`
+- `pin_right_rim`
+- `widen_mouth`
+- `maintain_mouth`
+- `probe_inside`
+- `insert_actor`
+- `retrieve`
+
+For cloth:
+- `lift_edge`
+- `separate_layer`
+- `stabilize_fold`
+- `maintain_lift`
+- `insert_actor`
+- `retrieve`
+
+Represent these as discrete proposal tokens or a macro head in `action_decoder.py`, then produce continuous chunk deltas conditioned on the chosen macro. The planner should shortlist across macro families first and refine within each family second. That prevents “all candidates are tiny perturbations around the same wrong idea”.
+
+In `models/planner.py`, add hard feasibility gates before utility aggregation. Do not let the planner prefer “retrieve now” if actor feasibility, hold quality, or support stability are below threshold. Use worst-step or CVaR-style penalties for reocclusion and collapse, rather than only mean penalties. These tasks fail on bad tails, not just on averages.
+
+Why this matters: the current planner is too dependent on easy local ranking. Real reveal-and-retrieve requires semantically different plans, not just slightly different noise vectors.
+
+### 4.6 Change the loss stack to supervise what actually matters
+
+Files to change:
+
+`train/losses.py`  
+`train/trainer.py` (if needed for logging)  
+Add new tests: `tests/test_candidate_ranking_loss.py`, `tests/test_phase_labels_not_action_only.py`, `tests/test_planner_gradient_flow.py`
+
+Exact changes:
+
+Reduce dependence on heuristic phase labels inferred from the current action chunk. That heuristic is acceptable for early bootstrapping, but it should not remain the main source of phase supervision. Prefer simulator-side phase or subgoal labels where available. If those are not reliable, phase should be a weak auxiliary, not a strong driver.
+
+Add pairwise or listwise ranking loss over candidate action chunks using actual rollout utility labels. These labels should come from simulated outcomes, not just from “teacher is first, noise is worse”.
+
+Add consistency losses:
+- predicted opening quality should correlate with rollout persistence
+- predicted reocclusion should correlate with actual collapse after release
+- predicted uncertainty should be calibrated against outcome uncertainty or visibility error
+
+Lower the relative weight of pure behavior cloning once ranking and rollout supervision are reliable. This project should not stay as BC-with-many-auxiliaries.
+
+## 5. Mandatory data-generation changes
+
+Files to change:
+
+`sim_reveal/dataset.py`  
+`sim_reveal/procedural_envs.py`  
+Add new tests: `tests/test_dataset_hard_negative_presence.py`, `tests/test_no_leak_with_new_labels.py`, `tests/test_teacher_audit.py`
+
+Exact changes:
+
+The dataset generation path must stop relying on teacher-plus-Gaussian-noise as the dominant source of planner candidates. Keep the teacher as one source, but add hard negative families that reflect actual task failures.
+
+Required negative families for all three tasks:
+
+1. premature retrieve: actor attempts retrieval before corridor and hold quality are sufficient;
+2. reveal-with-release: revealer creates an opening but fails to maintain it;
+3. over-disturbance: revealer opens aggressively but causes collapse or damage risk;
+4. wrong-side or wrong-edge reveal: the opening is created in a useless place;
+5. delayed actor entry: revealer holds too long and wastes time or destabilizes the scene;
+6. actor path through weak corridor: actor enters where access exists visually but not safely.
+
+Required task-specific negative families:
+
+For foliage:
+- swipe that increases visibility briefly but induces immediate reocclusion;
+- push direction that hides the target from the actor side;
+- gap on the wrong side of the target.
+
+For bag:
+- one-rim lift that slips instead of widening the mouth;
+- opening wide enough visually but not stable enough for actor insertion;
+- actor reaches through the fabric instead of through the aperture.
+
+For cloth:
+- lift too high and destroy fold structure;
+- lift the wrong layer;
+- retrieve path that drags clothing and unfolds the stack.
+
+The dataset should record candidate-level rollout outcomes for every candidate chunk:
+- success
+- reveal achieved
+- visibility AUC
+- hold persistence
+- reocclusion rate
+- disturbance cost
+- fold-preservation (cloth)
+- mouth aperture / hold quality (bag)
+- damage proxy / gap width (foliage)
+
+This candidate-level outcome table should be the source of planner labels.
+
+Also add a teacher audit report. The current teacher is a useful bootstrap, but it is not enough to assume it is good. The audit should compare the teacher against reveal-only, retrieve-only, no-hold, and random policy baselines on the current proxy suite.
+
+## 6. Small but mandatory engineering cleanups
+
+These changes do not change model quality directly, but they reduce evaluation ambiguity and future regressions.
+
+In `tests/conftest.py`, remove the hardcoded `/workspace/VLAarchtests/code/reveal_vla_bimanual` path. Replace it with a path derived from `Path(__file__).resolve()` so tests run anywhere.
+
+In `eval/run_rlbench_rollout_eval.py`, preserve richer episode traces. Save chosen macro mode, planner scores, confidence, predicted reocclusion, path recoveries, noop fallbacks, and whether support-mode conditioning was enabled.
+
+In `eval/run_reveal_benchmark.py`, stop using only the default 24 episodes for serious comparisons. Keep 24 as a smoke benchmark, but add a “serious” mode at 100 or 200 episodes per proxy.
+
+In `eval/run_reveal_benchmark.py`, explicitly report `chunk_commit_steps` and do not leave the main reveal benchmark at a commit horizon of zero by default. These tasks are not purely one-step reactive.
+
+In the eval reporting utilities, add bootstrap confidence intervals and paired-seed comparisons. The differences you care about are often a few percentage points. Unpaired noisy comparisons are not enough.
+
+## 7. Exact new tests to verify the implementation
+
+The current repo has contract tests. Keep them. Add the following behavioral tests.
+
+### 7.1 Geometry and fusion tests
+
+`tests/test_geometry_tokens_propagate.py`
+
+Construct a tiny batch with fixed RGB and depth. Modify only camera rotation. Verify that:
+1. `geometry_tokens` change,
+2. the fused scene representation changes when geometry is enabled,
+3. the fused scene representation does not change when geometry is disabled.
+
+`tests/test_camera_rotation_geometry.py`
+
+Use two cameras with identical translation and different rotation. Verify that the policy representation is rotation-sensitive after the geometry fix. This should fail on the current code and pass after the change.
+
+### 7.2 Spatial memory tests
+
+`tests/test_spatial_memory_occlusion_persistence.py`
+
+Use a scripted proxy sequence where the target is briefly visible, then fully occluded, then visible again. Verify that belief memory retains a localized target belief during occlusion and sharpens it after reappearance. This should test both persistence and uncertainty.
+
+`tests/test_memory_slot_write_gating.py`
+
+Feed a scene where only the opening region changes. Verify that only a minority of memory slots or cells update. This prevents global overwriting.
+
+`tests/test_reocclusion_memory_regression.py`
+
+Create a scripted “open then release” sequence. Verify that memory tracks reocclusion and that predicted hold quality declines.
+
+### 7.3 World-model tests
+
+`tests/test_world_model_null_rollout.py`
+
+Assert that `null_rollout` returns an exact or near-exact identity state and does not apply unintended updates.
+
+`tests/test_world_model_identity_rollout.py`
+
+Assert that `identity_rollout` preserves state across steps while leaving logging fields consistent.
+
+`tests/test_world_model_field_consistency.py`
+
+Roll out one deterministic proxy step and compare predicted next-step fields against simulator privileged fields. Enforce MAE thresholds per field, not only a single scalar.
+
+`tests/test_world_model_task_adapter.py`
+
+Use the same initial field state with different task embeddings. Verify that transitions differ in a consistent way. This catches dead task-conditioning code paths.
+
+### 7.4 Candidate and planner tests
+
+`tests/test_candidate_macro_coverage.py`
+
+Verify that the proposal generator returns at least one candidate from each required macro family when requested.
+
+`tests/test_planner_reocclusion_gating.py`
+
+Create a scripted case where one candidate retrieves immediately but causes opening collapse, and another candidate maintains the opening first. Verify that the planner picks the maintain-first plan.
+
+`tests/test_proposal_semantic_diversity.py`
+
+Do not measure diversity only by vector distance. Also verify macro-family diversity and rollout outcome diversity.
+
+### 7.5 Task-head tests
+
+`tests/test_task_conditioned_head_shapes.py`
+
+Verify output presence and shapes for all common fields and all task-specific fields.
+
+`tests/test_task_metric_monotonicity.py`
+
+Use small synthetic perturbations:
+- increase aperture in bag: `opening_quality` should increase;
+- increase canopy gap in foliage: `actor_feasibility` should increase;
+- over-lift cloth: `fold_preservation` should decrease.
+
+These are not full scientific tests, but they catch dead or miswired heads quickly.
+
+### 7.6 Dataset and leakage tests
+
+`tests/test_dataset_hard_negative_presence.py`
+
+Sample dataset items and verify that candidate sets contain hard negative families, not just teacher-centered noise.
+
+`tests/test_no_leak_with_new_labels.py`
+
+Extend the no-leak assertions to cover all new task-specific labels and maps. The proxy dataset must keep using rendered observations only on the input side.
+
+`tests/test_teacher_audit.py`
+
+Require the teacher to beat random, retrieve-only, and reveal-only on the proxy metrics. If the teacher itself is weak, the whole planner training signal is questionable.
+
+### 7.7 Scripted proxy behavior suite
+
+Add a new deterministic behavioral test suite, for example under `tests/test_proxy_scripted_bench.py`.
+
+This suite should include 10 to 20 deterministic seeds per task with hand-designed initial states. The expected winner should be obvious.
+
+Required scripted cases:
+- bag: `maintain_mouth` should beat `retrieve` immediately on hold persistence and success;
+- foliage: `pin_canopy` should beat `random_swipe` on reocclusion and visibility AUC;
+- cloth: `stabilize_fold` should beat `lift_high` on fold-preservation and success.
+
+The full model does not need to be perfect on these, but the planner should select the intended candidate at least 80 percent of the time.
+
+## 8. Exact benchmark plan to estimate performance
+
+Separate the benchmarks into two layers. The first layer verifies that the implementation behaves correctly. The second estimates real performance against baselines.
+
+### 8.1 Layer A: implementation-verification benchmarks
+
+These are not publication benchmarks. They are gates.
+
+Run the full unit and integration suite after every architecture milestone:
+
+```bash
+PYTHONPATH=code/reveal_vla_bimanual pytest tests -q
+```
+
+After the new behavioral tests are added, require all of the following before moving on:
+- all geometry propagation tests pass;
+- the scripted proxy suite passes;
+- world-model null and identity ablations pass exactly;
+- candidate macro coverage passes;
+- no-leak assertions pass with new task fields.
+
+Then run a deterministic proxy smoke benchmark on fixed seeds (for example 10 per task) to catch obvious regressions:
+
+```bash
+cd code/reveal_vla_bimanual
+python -m eval.run_reveal_benchmark \
+  --model full=/abs/path/checkpoint.pt \
+  --episodes 10 \
+  --proxies foliage bag cloth \
+  --chunk-commit-steps 4 \
+  --output-root /abs/path/reports/reveal_smoke
+```
+
+This benchmark is only for regression detection. It is not a performance claim.
+
+### 8.2 Layer B: strengthened proxy benchmark (main task-aligned benchmark now)
+
+This should become the main internal benchmark until real teleop data exists.
+
+Use the existing `foliage`, `bag`, and `cloth` proxies, but strengthen them and evaluate seriously:
+- at least 100 deterministic seeds per proxy for final comparisons;
+- paired-seed evaluation across all ablations;
+- chunk commit horizons of at least 4, and also report a 0/2/4 sweep once;
+- no teacher involvement during evaluation.
+
+Run the base benchmark:
+
+```bash
+cd code/reveal_vla_bimanual
+python -m eval.run_reveal_benchmark \
+  --model full=/abs/path/checkpoint.pt \
+  --episodes 100 \
+  --proxies foliage bag cloth \
+  --chunk-commit-steps 4 \
+  --output-root /abs/path/reports/reveal_full
+```
+
+Run required paired ablations from the same checkpoint family or retrained checkpoints:
+- no geometry tokens
+- no spatial memory
+- compact world model instead of spatial
+- no planner
+- planner with Gaussian candidates only
+- no task-conditioned head
+- no support-mode conditioning
+
+The proxy benchmark must report at least these metrics:
+- retrieve success
+- reveal success
+- target visibility AUC
+- actor-feasibility AUC
+- hold persistence
+- reocclusion rate
+- disturbance cost
+- planner top-1 on candidate rollouts
+- world-model next-step MAE
+- uncertainty calibration
+- candidate ranking NDCG
+
+Add task-specific metrics:
+- foliage: gap width, damage proxy, release-collapse rate
+- bag: aperture width or area, rim slip rate, insertion success
+- cloth: fold-preservation score, layer separation quality, drag-induced disturbance
+
+Acceptance gate for continuing toward public baseline comparison:
+- the full model should beat the current repo’s RGB-D baseline on mean proxy success and on at least two of the three proxies;
+- planner-on should beat planner-off on at least two of the three proxies and on hard-negative candidate ranking;
+- spatial world model should beat compact and null rollouts on persistence and reocclusion prediction;
+- task-conditioned head should beat generic head on at least one task-specific metric per target task.
+
+### 8.3 Layer C: RLBench / PerAct2 bimanual rollout benchmark
+
+The repo already has the right hook for this. Use `run_rlbench_rollout_eval.py` and `run_peract2_task_sweep.py` as the main public benchmark entry points. Do not treat `run_peract2_launch_smoke.py` as evaluation. It is only a launch check.
+
+Run the full existing PerAct2 13-task split from `sim_rlbench/task_splits.py::PERACT2_BIMANUAL_TASKS`:
+
+```bash
+cd code/reveal_vla_bimanual
+python -m eval.run_peract2_task_sweep \
+  --checkpoint /abs/path/checkpoint.pt \
+  --output-root /abs/path/reports/peract2_13 \
+  --episodes-per-task 25 \
+  --episode-length 20 \
+  --resolution 224 \
+  --chunk-commit-steps 4 \
+  --allow-unsupervised-planning \
+  --headless
+```
+
+Also run direct single-task evaluations when debugging:
+
+```bash
+cd code/reveal_vla_bimanual
+python -m eval.run_rlbench_rollout_eval \
+  --checkpoint /abs/path/checkpoint.pt \
+  --output-dir /abs/path/reports/rlbench_debug \
+  --tasks RightOpenDrawer \
+  --episodes-per-task 25 \
+  --episode-length 20 \
+  --resolution 224 \
+  --plan \
+  --chunk-commit-steps 4 \
+  --allow-unsupervised-planning \
+  --headless
+```
+
+This benchmark is not a direct match to the three target tasks, but it is the main public bimanual sanity check. It measures whether the structured modifications hurt or help general bimanual competence.
+
+Required comparisons on this benchmark:
+- current repo best checkpoint
+- full improved model
+- no-planner ablation
+- compact world model ablation
+- no geometry ablation
+- no task-conditioning ablation
+
+If external baseline code is available, evaluate against:
+- PerAct2
+- InterACT
+- VoxAct-B
+- AnyBimanual
+
+If compute allows, also compare against foundation-scale baselines as a separate category:
+- TwinVLA
+- RDT-1B
+
+Fairness requirements:
+- same camera setup if possible (front plus both wrists);
+- same resolution;
+- same episode length and reset policy;
+- same task list;
+- same number of evaluation episodes;
+- report whether baselines use extra large-scale pretraining.
+
+This benchmark should report:
+- per-task success
+- mean success
+- mean return
+- path recoveries
+- noop fallbacks
+- plan-on vs plan-off
+- per-episode planner traces for error analysis
+
+### 8.4 Layer D: deformable-manipulation public benchmarks
+
+You do not yet have custom teleop data, so the closest public matches for bag and cloth should be used now.
+
+Recommended benchmarks:
+- DeformableRavens
+- SoftGym cloth tasks
+- DaXBench cloth tasks
+
+The exact subset should be chosen based on available tasks, but the mapping is straightforward. Bag-like opening and insertion tasks are the closest public proxy for the bag environment. Cloth lifting, separation, and manipulation tasks are the closest public proxy for the suitcase environment. There is no equally good public foliage benchmark, so the strengthened foliage proxy will remain the main stand-in until custom data exists.
+
+Required evaluation protocol:
+- same observation modalities across methods;
+- same action horizon where possible;
+- same random seeds;
+- same episode budgets;
+- report both success and task-specific deformation metrics.
+
+Add at least these extra metrics on the deformable benchmarks:
+- opening quality or aperture quality
+- hold persistence under actor motion
+- reocclusion or collapse rate
+- disturbance cost
+- fold-preservation or structural-preservation score
+
+### 8.5 Layer E: optional exploratory / active-perception benchmark
+
+If EFM-10 or BAP code and data are actually available when implementation starts, add them. That benchmark is conceptually close to your task family because it measures exploratory plus focused manipulation under occlusion. Do not block the project on it if code is not readily usable.
+
+### 8.6 Layer F: optional broad generalization benchmark
+
+If time allows, add RoboTwin 2.0 as a general bimanual breadth check. It is not a direct target-task match, but it is useful for checking whether the structured reveal-and-retrieve bias damages general bimanual transfer.
+
+## 9. Baseline strategy
+
+There are two baseline groups and they should not be mixed carelessly.
+
+The first group is matched-data or matched-setting baselines. These are the most useful for fair engineering comparison. Use PerAct2, InterACT, VoxAct-B, and AnyBimanual if code is available in a compatible evaluation setting.
+
+The second group is foundation-scale baselines. These are useful, but they are not apples-to-apples unless you disclose the pretraining and model scale difference clearly. Use TwinVLA and RDT-1B in this category if compute allows.
+
+Do not declare victory because the improved model beats the current repo checkpoint. That is a necessary condition, not the target claim.
+
+## 10. Acceptance criteria for “ready to collect real data”
+
+Do not move into expensive teleop collection until all of the following are true.
+
+First, the geometry and spatial memory tests pass and stay green for multiple checkpoints.
+
+Second, the strengthened proxy benchmark shows that the full model beats the current repo baseline convincingly. The minimum bar should be improvement in overall proxy success plus improvement on at least two of the three task types.
+
+Third, planner-on must beat planner-off on hard-negative ranking and on task success. If the planner does not beat the decoder baseline, then the explicit planning stack is not yet earning its complexity.
+
+Fourth, the spatial world model must beat compact and null baselines on persistence and reocclusion prediction. If it does not, the planning story is still too weak.
+
+Fifth, the improved model should at least match strong public baselines on the RLBench / PerAct2 suite, and ideally exceed them on the tasks most related to opening, holding, uncovering, and coordinated retrieval. If it is significantly behind there, the architecture is still too immature.
+
+## 11. Recommended implementation order
+
+Phase 1 should fix information flow and evaluation trustworthiness. Implement geometry propagation, camera orientation encoding, and path cleanup in `tests/conftest.py`. Then add the new geometry tests and rerun the current proxy benchmark.
+
+Phase 2 should add task-aware semantic candidates and hard-negative data generation. This is the fastest path to making the planner meaningful without yet rewriting the full memory and world model stack.
+
+Phase 3 should add task-conditioned reveal outputs and the strengthened proxy metrics. At this stage the proxy benchmark should start reflecting the real task failure modes.
+
+Phase 4 should replace pooled memory and compact rollout with the new spatial memory and spatial world model. This is the biggest change and should only happen after the eval harness can tell whether it helped.
+
+Phase 5 should run the full internal ablation suite, then RLBench / PerAct2, then deformable public benchmarks, and only then decide whether the architecture is strong enough to justify real-data collection.
+
+## 12. What to avoid
+
+Do not treat launch smoke as performance evaluation.
+
+Do not keep teacher-centered Gaussian candidates as the main planner supervision source.
+
+Do not remove task structure in favor of a generic monolithic BC model unless the structured architecture clearly fails. Nothing in the current repo proves that.
+
+Do not use only mean success. These tasks need persistence, reocclusion, and structural-preservation metrics.
+
+Do not claim the current planner or current world model are validated. They are not, yet.
+
+## 13. Minimal first patch set (the first pull request)
+
+If only one implementation sprint is possible before deeper refactors, the first pull request should contain exactly this:
+
+1. fix `geometry_tokens` propagation from backbone to fusion to policy output;
+2. add camera rotation encoding in `DepthPatchAdapter`;
+3. add `tests/test_geometry_tokens_propagate.py` and `tests/test_camera_rotation_geometry.py`;
+4. replace hardcoded path logic in `tests/conftest.py`;
+5. extend `run_reveal_benchmark.py` reporting to save `chunk_commit_steps`, bootstrap confidence intervals, and paired-seed summaries;
+6. add semantic macro candidates in `action_decoder.py` without yet deleting the Gaussian fallback;
+7. add hard negative candidate generation in `sim_reveal/procedural_envs.py`;
+8. add the deterministic scripted proxy benchmark suite.
+
+This first patch set will not make the model SOTA. It will make the repo trustworthy enough to support the larger refactor.
+
+## 14. Reference links
+
+Repo root:  
+https://huggingface.co/lsnu/VLAarchtests/tree/main
+
+Core files:  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/backbones.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/multiview_fusion.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/observation_memory.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/reveal_head.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/world_model.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/action_decoder.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/planner.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/models/policy.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/train/losses.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/sim_reveal/dataset.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/sim_reveal/procedural_envs.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/eval/run_reveal_benchmark.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/eval/run_rlbench_rollout_eval.py  
+https://huggingface.co/lsnu/VLAarchtests/blob/main/code/reveal_vla_bimanual/eval/run_peract2_task_sweep.py
+
+Public benchmark / baseline references to align against:  
+PerAct2 / RLBench2 bimanual benchmark: https://bimanual.github.io/  
+InterACT: https://dannyran123.github.io/interact/  
+VoxAct-B: https://voxact-b.github.io/  
+AnyBimanual: https://anybimanual.github.io/  
+TwinVLA: https://twinvla.github.io/  
+RDT-1B: https://rdt-robotics.github.io/rdt-robotics/  
+DeformableRavens: https://deformableravens.github.io/  
+SoftGym: https://sites.google.com/view/softgym/home  
+DaXBench: https://daxbench.github.io/  
+EFM / BAP: https://efmanipulation.github.io/  
+RoboTwin 2.0: https://robotwin-platform.github.io/
+
+## 15. Final recommendation
+
+The architecture should be pursued, but only in a narrower and more explicit form: task-structured bimanual reveal-and-retrieve under elastic occlusion. The current repo is close enough to that idea to be worth continuing. The most important next step is not collecting real data yet. It is making the geometry path real, making the planner learn from hard failure cases, and making the world model spatial enough that “maintain the opening while the other arm retrieves” is something the system can actually predict rather than merely imitate.

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons/command.txt

@@ -0,0 +1 @@
+/workspace/envs/rlbench/bin/python -m eval.run_rlbench_rollout_eval --checkpoint /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt --output-dir /workspace/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons --tasks bimanual_dual_push_buttons --episodes-per-task 1 --episode-length 20 --resolution 224 --device cuda --chunk-commit-steps 4 --headless --plan --allow-unsupervised-planning

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons/rollout_eval.json

@@ -0,0 +1,333 @@
+{
+  "checkpoint": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+  "plan_requested": true,
+  "plan_applied": true,
+  "planner_mode": "trainable",
+  "support_mode_conditioning": true,
+  "task_conditioning": true,
+  "geometry_enabled": true,
+  "world_model_mode": "checkpoint_default",
+  "episodes_per_task": 1,
+  "episode_length": 20,
+  "resolution": 224,
+  "reset_retries": 20,
+  "cameras": [
+    "front",
+    "wrist_left",
+    "wrist_right"
+  ],
+  "tasks": {
+    "bimanual_dual_push_buttons": {
+      "task_class": "BimanualDualPushButtons",
+      "successes": [
+        0.0
+      ],
+      "returns": [
+        0.0
+      ],
+      "path_recoveries": [
+        0
+      ],
+      "noop_fallbacks": [
+        0
+      ],
+      "reset_retries": [
+        0
+      ],
+      "episode_traces": [
+        {
+          "language_goal": "push the olive and the orange buttons",
+          "steps": [
+            {
+              "timestep": 0,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 1,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 2,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 3,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 4,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 5,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 6,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 7,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 8,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 9,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 10,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 11,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 12,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 13,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 14,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 15,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 16,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 17,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 18,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 19,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            }
+          ],
+          "success": 0.0,
+          "return": 0.0,
+          "path_recoveries": 0,
+          "noop_fallbacks": 0
+        }
+      ],
+      "mean_success": 0.0,
+      "mean_return": 0.0
+    }
+  },
+  "mean_success": 0.0
+}

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons/rollout_eval.md

@@ -0,0 +1,14 @@
+# RLBench Rollout Eval
+
+- Checkpoint: `/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt`
+- Plan requested: `True`
+- Plan applied: `True`
+- Support-mode conditioning: `True`
+- Task conditioning: `True`
+- Geometry enabled: `True`
+- World-model mode: `checkpoint_default`
+- Mean success: `0.000`
+
+## Per-task
+
+- `bimanual_dual_push_buttons`: mean_success=0.000, returns=[0.0]

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons/stderr.txt

@@ -0,0 +1,4 @@
+/workspace/envs/rlbench/lib/python3.10/site-packages/torch/nn/modules/transformer.py:306: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
+  warnings.warn(f"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}")
+qt.qpa.xcb: QXcbConnection: XCB error: 148 (Unknown), sequence: 181, resource id: 0, major code: 140 (Unknown), minor code: 20
+WARNING:root:not sure how _robot_shapes are used is used.

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_dual_push_buttons/stdout.txt

@@ -0,0 +1,334 @@
+{
+  "checkpoint": "/workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt",
+  "plan_requested": true,
+  "plan_applied": true,
+  "planner_mode": "trainable",
+  "support_mode_conditioning": true,
+  "task_conditioning": true,
+  "geometry_enabled": true,
+  "world_model_mode": "checkpoint_default",
+  "episodes_per_task": 1,
+  "episode_length": 20,
+  "resolution": 224,
+  "reset_retries": 20,
+  "cameras": [
+    "front",
+    "wrist_left",
+    "wrist_right"
+  ],
+  "tasks": {
+    "bimanual_dual_push_buttons": {
+      "task_class": "BimanualDualPushButtons",
+      "successes": [
+        0.0
+      ],
+      "returns": [
+        0.0
+      ],
+      "path_recoveries": [
+        0
+      ],
+      "noop_fallbacks": [
+        0
+      ],
+      "reset_retries": [
+        0
+      ],
+      "episode_traces": [
+        {
+          "language_goal": "push the olive and the orange buttons",
+          "steps": [
+            {
+              "timestep": 0,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 1,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 2,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 3,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.197153091430664,
+                11.241825103759766,
+                11.236907005310059,
+                11.205011367797852
+              ],
+              "predicted_reocclusion": 0.5305227041244507,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 4,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 5,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 6,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 7,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.1263427734375,
+                11.163692474365234,
+                11.160633087158203,
+                11.130797386169434
+              ],
+              "predicted_reocclusion": 0.5315501689910889,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 8,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 9,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 10,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 11,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.077136039733887,
+                11.114724159240723,
+                11.111690521240234,
+                11.081738471984863
+              ],
+              "predicted_reocclusion": 0.5313586592674255,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 12,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 13,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 14,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 15,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.042268753051758,
+                11.08004379272461,
+                11.07697868347168,
+                11.046956062316895
+              ],
+              "predicted_reocclusion": 0.5312807559967041,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 16,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 17,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 18,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            },
+            {
+              "timestep": 19,
+              "chosen_macro_mode": "lift_support_layer",
+              "planner_scores": [
+                11.03925895690918,
+                11.076944351196289,
+                11.073898315429688,
+                11.043900489807129
+              ],
+              "predicted_reocclusion": 0.5312473773956299,
+              "support_mode_conditioning": true,
+              "path_recoveries": 0,
+              "noop_fallbacks": 0
+            }
+          ],
+          "success": 0.0,
+          "return": 0.0,
+          "path_recoveries": 0,
+          "noop_fallbacks": 0
+        }
+      ],
+      "mean_success": 0.0,
+      "mean_return": 0.0
+    }
+  },
+  "mean_success": 0.0
+}
+[CoppeliaSim:loadinfo]   done.

results/2026-03-25-runpod/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_handover_item/command.txt

@@ -0,0 +1 @@
+/workspace/envs/rlbench/bin/python -m eval.run_rlbench_rollout_eval --checkpoint /workspace/VLAarchtests/artifacts/outputs/r3d/proxy_interaction_r3d_stage3_clip_rgbd_seed17/checkpoint_best.pt --output-dir /workspace/reports/peract2_baseline_ep1/baseline_rgbd_seed17_plan_split/bimanual_handover_item --tasks bimanual_handover_item --episodes-per-task 1 --episode-length 20 --resolution 224 --device cuda --chunk-commit-steps 4 --headless --plan --allow-unsupervised-planning