handoff/instructions4.md

# Developer handoff: 10-hour sim sprint on 1×RTX PRO 6000 for elastic-occlusion bimanual reveal/retrieve

## Scope

This handoff is for the current simulation phase only. The purpose is not to produce publication-grade evidence. The purpose is to use one short sprint to extract the most decision-relevant signal possible before the custom three-task teleoperation benchmark exists.

This document does **not** include explicit instructions for future teleoperation data collection.

The target problem remains the same: bimanual reveal and retrieve under elastic occlusion, with task families that map to (1) foliage reveal with safe actor insertion and retrieval, (2) bag opening plus retrieval, and (3) folded-cloth / suitcase reveal with minimal fold disruption.

## Hard constraints for this sprint

The sprint must assume the following:

- Hardware: **1× RTX PRO 6000** workstation GPU.
- Wall-clock budget: **~10 hours total**.
- Deliverable standard: **decision-quality results**, not paper-quality results. They must be rigorous nonetheless - absolutely no data leaks.
- The output of the sprint must be enough to shape the next development cycle.

This means the sprint is not allowed to expand into a broad refactor, a large hyperparameter search, an external benchmark integration project, or a foundation-model migration. The objective is to get the strongest signal per hour.

## What this sprint needs to answer

At the end of the 10-hour window, the repo should let us answer these questions with reasonable confidence:

1. Does the current full architecture look directionally better than trivial and structured baselines on rough proxy versions of the three real task families?
2. Which parts of the architecture appear to matter most right now: explicit task conditioning, geometry, memory, planner, or candidate family structure?
3. For each of the three target tasks, is the current architecture best described as **promising**, **uncertain**, or **weak** under the proxy tests?
4. If the system is weak, is the weakness coming mostly from perception/state estimation, memory, retrieve gating/planning, or proxy mismatch?
5. Which next engineering changes are most likely to widen the eventual real-task performance gap, and which ones are unlikely to matter?

That is the bar for success in this sprint. It is acceptable if the results are approximate. It is not acceptable if the results are too noisy or too poorly instrumented to support those decisions.

## What this sprint can and cannot tell us

This sprint **can** tell us whether the current structured architecture is showing the right dependencies under stress. For example, it can tell us whether memory actually helps under reocclusion, whether geometry actually helps under camera perturbation, whether retrieve gating blocks premature retrieve, and whether the planner adds value beyond trivial candidate selection.

This sprint **cannot** tell us true real-world performance on live foliage, real bag interiors, or folded clothes in a suitcase. It also cannot tell us whether the final production backbone should be CLIP, OpenVLA-style, LingBot-style, or something else. Those are later decisions.

So the correct target is not “exact future performance.” The correct target is “a useful approximation of whether the current structure is pointed in the right direction, and where the biggest current bottlenecks are.”

## High-level decisions to lock now

1. Treat the current **compact-phase CLIP/RGB-D handoff branch** as the only reference branch for this sprint.
2. Keep the explicit reveal-state stack. Do **not** rewrite the repo into a monolithic end-to-end VLA policy in this sprint.
3. Keep RLBench only as a smoke test for three-camera and bimanual integration. Do **not** use RLBench mean success as the main selector for reveal/retrieve architecture changes.
4. Do **not** switch to a new backbone in this sprint. The goal here is to evaluate the structure, not to spend the 10-hour budget on trunk migration.
5. Prefer **eval-time knockouts and toggles** over fully retrained matched ablations in this sprint. Retrained matched ablations remain important later, but they are too expensive for the current time budget.

## Immediate read of the current repo

The current repo is still a good scaffold, but the signal quality is weaker than the structure quality. The strongest part of the system remains the decomposition itself: explicit reveal-state fields, dual scene/belief memory, task-conditioned proposal families, and a planner that reasons about persistence, reocclusion, support, and actor feasibility. The weakest part is that the current evidence is still too easy to misread. The proxy results already suggest that the large spatial rollout path is not the right place to spend another iteration right now, while the compact-phase line remains the most credible base. That should be treated as settled for this sprint.

## Sprint strategy

The fastest path to useful conclusions is:

1. Remove the most obvious confounds.
2. Add just enough stress slicing and logging to make the proxy benchmark informative.
3. Run a **small but fixed stratified benchmark** that is reused for every comparison.
4. Compare the base model against trivial baselines and a few **eval-time architecture knockouts**.
5. Use the results to build a task-by-task bottleneck map.

This is deliberately narrower than the earlier broad handoff. The point is to obtain meaningful conclusions within one GPU and one workday.

## Do now vs defer

### Must do in this sprint

These are the changes that are worth the time even under a 10-hour cap.

1. Explicit task metadata must override text routing.
2. History camera geometry must propagate correctly.
3. The compact-phase branch must become the default base config.
4. The proxy benchmark must become stratified by task and stress slice.
5. The benchmark runner must support simple baselines and eval-time architecture knockouts.
6. Reporting must include the small set of task-specific metrics that actually matter for the three target tasks.

### Explicitly defer until after this sprint

These are good ideas, but they should not consume the current 10-hour budget.

1. Foundation trunk migration (OpenVLA, LingBot, π0.5, etc.).
2. External deformable benchmark integration.
3. Full matched retraining ablation suite.
4. Major loss redesign or long retraining campaigns.
5. Large nuisance sweeps beyond the narrow stress slices listed below.
6. Spatial rollout branch rescue work.

## Mandatory repo changes for this sprint

### 1. Replace heuristic task routing as the primary path

**Why now:** this is a real confound and cheap to fix.

Files to edit:

- `code/reveal_vla_bimanual/models/policy.py`
- `code/reveal_vla_bimanual/models/action_decoder.py`
- `code/reveal_vla_bimanual/sim_reveal/dataset.py`
- `code/reveal_vla_bimanual/sim_reveal/generate_dataset.py`
- `code/reveal_vla_bimanual/eval/run_reveal_benchmark.py`

Required changes:

- Add `task_name` and `task_id` to every proxy training and evaluation example.
- Make explicit task metadata override any text-based inference everywhere.
- Keep keyword routing only as a fallback for legacy examples that do not carry task metadata.
- Surface the resolved task family in benchmark logs so mistakes are easy to see.

Acceptance criterion:

- A misleading prompt string must not change the task family when `task_name` is present.

### 2. Fix history geometry propagation

**Why now:** if geometry is broken in history, the current geometry ablations are not trustworthy.

Files to edit:

- `code/reveal_vla_bimanual/models/policy.py`
- any history batching utility that currently drops camera matrices
- proxy dataset serialization if history camera metadata is missing

Required changes:

- Save and batch history camera intrinsics and extrinsics.
- Pass them through the history encoder when geometry and camera-pose tokens are enabled.
- Add a validity mask if some history frames do not have full camera metadata.
- Add a debug log or assertion path that makes it obvious whether history geometry is really being used.

Acceptance criterion:

- A geometry-enabled run must receive non-null history camera tensors in the forward path.

### 3. Freeze the compact-phase branch as the main base

**Why now:** the time budget does not allow another architecture round on the weaker spatial branch.

Files to edit:

- `code/reveal_vla_bimanual/train/configs/*.yaml`
- `code/reveal_vla_bimanual/eval/run_ablations.py`
- any training or eval launcher that still points to older dummy ablation configs

Required changes:

- Create one new base config derived from the compact-phase recipe. Suggested filename:
  - `proxy_interaction_r3d_stage3_clip_rgbd_handoff_compact_phase_v7_base.yaml`
- Mark the current spatial configs as experimental.
- Make the new v7 base the default for this sprint.
- Do not create a broad new family of retrained ablation configs now. That is for later.

Acceptance criterion:

- The benchmark runner should use the v7 compact-phase config by default unless explicitly told otherwise.

### 4. Add a narrow but informative proxy stress suite

**Why now:** the current proxies are too undifferentiated. The sprint only needs enough stress structure to make results interpretable.

Files to edit:

- `code/reveal_vla_bimanual/sim_reveal/procedural_envs.py`
- `code/reveal_vla_bimanual/sim_reveal/proxy_specs.py`
- `code/reveal_vla_bimanual/sim_reveal/dataset.py`
- `code/reveal_vla_bimanual/sim_reveal/generate_dataset.py`

Required changes:

Add only these stress slices for this sprint:

- `nominal`
- `high_reocclusion`
- `camera_perturbation`
- one task-specific critical slice per task:
  - foliage: `tight_corridor_high_collateral`
  - bag: `one_sided_slip`
  - cloth: `fold_sensitive_long_persistence`

Also add:

- `difficulty_bin` with only `medium` and `hard` for this sprint (skip easy and extreme to save time and focus on decision-relevant cases)
- episode metadata for the sampled nuisance parameters used in each stress slice
- per-step traces for visibility, support, access/corridor, reocclusion risk, disturbance, and chosen candidate family

Acceptance criterion:

- Every benchmark report must be sliceable by task family, stress slice, and difficulty bin.

### 5. Add simple baselines and oracle-style planner evaluation

**Why now:** without these, base-model numbers are hard to interpret.

Files to edit/add:

- `code/reveal_vla_bimanual/eval/run_reveal_benchmark.py`
- `code/reveal_vla_bimanual/eval/metrics.py`
- add `code/reveal_vla_bimanual/eval/run_proxy_random_eval.py`
- add `code/reveal_vla_bimanual/eval/run_proxy_candidate0_eval.py`
- add `code/reveal_vla_bimanual/eval/run_planner_oracle_eval.py`
- add `code/reveal_vla_bimanual/eval/run_proxy_scripted_eval.py` if the existing scripted path is not already callable directly

Required changes:

- Add random candidate selection.
- Add candidate-0 selection.
- Add scripted teacher execution.
- Add oracle-planner evaluation that uses proxy candidate summaries directly.
- Add support for eval-time architecture toggles:
  - `--disable_planner`
  - `--disable_memory`
  - `--disable_task_conditioning`
  - `--disable_geometry`
  - `--disable_camera_pose` (optional if it is cheap)

Acceptance criterion:

- The benchmark must be able to compare the same checkpoint against trivial baselines and against architecture knockouts without retraining all variants.

### 6. Strengthen reporting, but only where it matters

**Why now:** the current sprint only succeeds if it produces conclusions, not just numbers.

Files to edit:

- `code/reveal_vla_bimanual/eval/metrics.py`
- `code/reveal_vla_bimanual/eval/run_reveal_benchmark.py`

Required metric outputs:

Global:

- overall proxy success
- per-task success
- success by stress slice
- success by difficulty bin
- premature retrieve rate
- reocclusion-after-reveal rate
- planner regret (where oracle summaries are available)

Task-specific headline metrics:

Foliage:

- visibility integral
- corridor availability
- collateral motion / damage proxy
- actor-feasibility floor before retrieve

Bag:

- mouth aperture
- hold persistence
- rim slip rate
- insertable corridor

Cloth:

- fold preservation
- layer separation quality
- top-layer stability
- lift-too-high rate

Required report shape:

- one overall table
- one task × stress slice table
- per-episode JSON traces for failure clustering later

Acceptance criterion:

- A single report should make it obvious whether the model is failing because of reocclusion, geometry sensitivity, premature retrieve, or task-specific degradation.

## Changes that are useful only if they are cheap

These are allowed only if the mandatory work finishes early.

### A. Light training rebalance

Only do this if it can be implemented in under about one hour.

Allowed small changes:

- oversample obvious hard negatives already present in the proxy dataset
- slightly increase loss weight on unsafe-retrieve ranking errors
- log candidate ranking diagnostics during training

Do **not** do a broad loss redesign in this sprint.

### B. Trunk modularity prep

Only do this if the mandatory work is already complete.

Allowed small changes:

- define a simple trunk adapter interface around the current CLIP path
- avoid touching planner, memory, or reveal-head code

Do **not** attempt a real new-backbone integration in this sprint.

## 10-hour execution plan

This schedule is the intended wall-clock plan. It is aggressive but realistic if the scope stays narrow.

### Hour 0 to 1.5: remove confounds

Complete:

- explicit task metadata path
- history geometry propagation
- v7 compact-phase base config
- eval-time toggle plumbing in the benchmark runner

Output expected by the end of this block:

- code compiles
- a tiny smoke run confirms task routing and history geometry are active

### Hour 1.5 to 3: build the fixed eval set and reporting

Complete:

- stratified proxy eval set generation
- task/stress/difficulty metadata roundtrip
- benchmark tables and per-episode JSON traces
- random, candidate-0, scripted, and oracle evaluation entry points

Output expected by the end of this block:

- one fixed benchmark episode set reused by every later run

### Hour 3 to 5.5: produce one base-model result

Preferred path:

- reuse the best existing compact-phase checkpoint if it still loads cleanly after the metadata and geometry fixes
- if needed, run a short warm-start fine-tune from that checkpoint rather than starting from scratch

Do not spend this block on multi-seed training. One strong base run is more valuable than several weak incomplete runs.

Output expected by the end of this block:

- one evaluated base model on the full fixed proxy suite

### Hour 5.5 to 8.5: run baselines and eval-time knockouts

Required comparisons:

- random
- candidate-0
- scripted teacher
- oracle planner
- base model
- base model with planner disabled
- base model with memory disabled
- base model with task conditioning disabled
- base model with geometry disabled
- base model with camera pose disabled if cheap enough

Output expected by the end of this block:

- a complete comparison table on the same episodes

### Hour 8.5 to 10: summarize and extract conclusions

Complete:

- task-by-task bottleneck summary
- approximate transfer-readiness labels for foliage, bag, and cloth
- ranked next-step engineering priorities

Output expected by the end of this block:

- one benchmark summary table
- one short conclusion memo in the repo or artifact directory

## Fixed benchmark design for the 10-hour sprint

Use one small, fixed, stratified benchmark. Reuse the same episode seeds for all variants.

Recommended size:

- **300 total episodes**
- **100 per task family**
- per task family:
  - 20 `nominal` / `medium`
  - 20 `nominal` / `hard`
  - 20 `high_reocclusion`
  - 20 `camera_perturbation`
  - 20 task-specific critical slice

This is small enough to run repeatedly and large enough to show directional differences if the logging is good.

## Required tests to add now

The test suite in this sprint is not meant to be exhaustive. It is meant to prevent false conclusions.

### Unit tests

`tests/test_explicit_task_metadata_overrides_text.py`

- Batch has `task_name="bag"` and misleading foliage text.
- Assert that bag proposal families and bag task heads are used.

`tests/test_text_routing_only_used_as_fallback.py`

- Assert that keyword routing is skipped when task metadata exists.

`tests/test_history_camera_geometry_propagates.py`

- Assert that history frames receive non-null camera tensors when geometry is enabled.

`tests/test_history_geometry_validity_mask.py`

- Assert that mixed valid/invalid history geometry uses a validity mask rather than silent nulling.

`tests/test_eval_toggle_paths_work.py`

- Assert that planner, memory, task-conditioning, and geometry toggles actually change the execution path.

`tests/test_benchmark_report_contains_task_and_stress_slices.py`

- Assert that the output report includes task family, stress slice, and difficulty bin tables.

### Integration tests

`tests/test_proxy_stress_profile_metadata_roundtrip.py`

- Generate all sprint stress slices and assert the metadata survives dataset serialization and evaluation.

`tests/test_planner_beats_random_on_oracle_candidates.py`

- Use oracle candidate summaries and assert the planner beats random and candidate-0 on regret/top-1.

`tests/test_memory_matters_under_high_reocclusion.py`

- Compare full memory vs disabled-memory on a small high-reocclusion slice and assert a directional drop.

`tests/test_geometry_matters_under_camera_perturbation.py`

- Compare geometry-on vs geometry-off on a small camera-perturbation slice and assert a directional drop.

`tests/test_retrieve_gating_blocks_premature_retrieve.py`

- Feed candidates where raw retrieve looks tempting but support/persistence/corridor are unsafe and assert that retrieve is rejected.

These tests are enough for this sprint. Do not expand the suite unless the mandatory implementation is already done.

## Benchmark runs to perform in this sprint

### 1. Baselines

Run on the fixed 300-episode benchmark:

- random candidate selection
- candidate-0 selection
- scripted teacher
- oracle planner

These establish the floor, the structured upper bound for the current proposal set, and whether the learned planner adds anything.

### 2. Base model run

Run the v7 compact-phase base on the same 300 episodes.

### 3. Eval-time architecture knockouts

Run the same checkpoint with these toggles:

- no planner
- no memory
- no task conditioning
- no geometry
- no camera pose (only if cheap)

These are not final scientific ablations. They are fast directional probes that tell us what appears to matter right now.

### 4. Optional cheap extra run

Only if time remains:

- a short warm-start fine-tune with a small hard-negative rebalance

Treat this only as bonus signal. It is not part of the minimum sprint success condition.

## Decision rubric for interpreting the results

At the end of the sprint, do **not** only report the raw numbers. Convert the results into a bottleneck map.

### Signals that the architecture is directionally healthy

- Base clearly beats random and candidate-0 on all three task families.
- Oracle planner beats random and candidate-0 by a wide margin, showing the proposal/planner structure is at least usable.
- Disabling planner hurts most on premature-retrieve and task-specific stress slices.
- Disabling memory hurts most on `high_reocclusion` and long-persistence cloth cases.
- Disabling geometry hurts most on `camera_perturbation`.
- Task conditioning matters on the mixed-task benchmark and under misleading text.

### Signals that a component is currently unproven

- Base is only slightly above candidate-0 or random.
- Oracle planner is weak, which means the proposal set or planner utility is not yet reliable.
- Memory removal is almost flat on `high_reocclusion`.
- Geometry removal is almost flat on `camera_perturbation`.
- Planner removal is flat, which suggests that the learned scores or the candidate shortlist are not carrying useful structure.

### Task-specific interpretation

For foliage, judge transfer-readiness mainly from:

- corridor availability
- actor-feasibility floor before retrieve
- collateral motion / damage proxy
- robustness under `high_reocclusion` and `tight_corridor_high_collateral`

For bag, judge transfer-readiness mainly from:

- mouth aperture
- hold persistence
- rim slip rate
- robustness under `one_sided_slip`

For cloth, judge transfer-readiness mainly from:

- fold preservation
- layer separation quality
- top-layer stability
- lift-too-high rate
- robustness under `fold_sensitive_long_persistence`

At the end of the sprint, label each task family as:

- **Promising**: base beats weak baselines clearly and the expected architectural components matter on the right stress slices.
- **Uncertain**: base is somewhat above baselines but at least one critical stress slice or component dependency is weak.
- **Weak**: base is near trivial baselines or the critical stress slices fail badly enough that the current structure is not yet convincing.

## Practical GPU/runtime guidance

This sprint assumes a single 96 GB workstation GPU. That is enough for the current CLIP-based compact-phase line and repeated proxy evaluations, but it is not enough time to justify broad parallel experiments.

Use the following operating rules:

- run experiments sequentially
- prefer one strong base run over many partial runs
- use mixed precision if already supported
- keep evaluation batch sizes modest and stable
- avoid large retraining loops or many seeds
- reuse the same fixed benchmark episodes for every comparison

The main bottleneck in this sprint should be engineering and interpretation, not raw VRAM.

## Things not to do in this sprint

Do not switch backbones. Do not integrate an external simulator. Do not rescue the spatial rollout branch. Do not run broad hyperparameter sweeps. Do not attempt a five-seed retraining ablation matrix. Do not use RLBench averages as the main argument for or against architecture changes meant for foliage, bag, or folded cloth.

## Minimal deliverables at the 10-hour mark

At the end of the sprint, the repo or artifact directory should contain:

1. the new `v7` compact-phase base config
2. the fixed 300-episode stratified benchmark definition or metadata file
3. updated benchmark runner with stress-slice reporting
4. random, candidate-0, scripted, and oracle evaluation runners
5. the required unit and integration tests listed above
6. one complete result table comparing:
   - random
   - candidate-0
   - scripted teacher
   - oracle planner
   - base model
   - no planner
   - no memory
   - no task conditioning
   - no geometry
   - optional no camera pose
7. one short decision memo that states:
   - approximate transfer-readiness for foliage, bag, and cloth
   - which architectural components look most important
   - which current weakness is most likely to block real-task success
   - what should be strengthened next

## Success condition for this sprint

This sprint is successful if, by the end of 10 hours, we can say something like the following and defend it with benchmark evidence:

- “The structured architecture appears meaningfully better than trivial baselines on foliage and bag proxies, but cloth remains fragile because fold preservation degrades under long persistence.”
- or “The planner structure looks sound under oracle candidates, but the learned state estimate is still too weak, so the next work should target perception/memory rather than planner redesign.”
- or “Geometry and task conditioning matter, but memory does not yet move the reocclusion slice, so the current memory story is still unproven.”

If we can make claims of that form with actual run outputs, the sprint has done its job.