README.md

---
tags:
  - robotics
  - vision-language-action
  - bimanual-manipulation
  - maniskill
  - rlbench
  - rgbd
---

# VLAarchtests4

`VLAarchtests4` is the fresh organization repo for the RunPod work staged from `/workspace` on `2026-04-01 UTC`.

It carries forward the earlier repo lineage and adds the current public-sim benchmark package work:

- `VLAarchtests`
  - early proxy + RLBench architecture search, handoff checkpoints, and environment recreation files from the `2026-03-25/26` sessions
- `VLAarchtests2`
  - larger exploratory organization repo with more baselines, overlap/anchor work, frequent model changes, mixed artifacts, and several results that required later reinterpretation
- `VLAarchtests3`
  - cleaned export focused on the elastic-occlusion `trunk + structured adapter + no-op fallback` refactor, validated tests, current checkpoints, and handoff docs
- `VLAarchtests4`
  - keeps the `VLAarchtests3` export intact and adds the full current workspace `reports/`, `outputs/`, and `data/` trees, including all public benchmark smoke runs, checkpoint directories, dataset bundles, validation sweeps, and environment snapshots from the public-sim evaluation pass

## What This Repo Adds

The main new addition in this repo is the public benchmark track work for the elastic-occlusion adapter:

- real public-sim smoke runs on:
  - ManiSkill `PickClutterYCB-v1` as the dense occluded retrieval proxy
  - ManiSkill bridge basket retrieval proxy as the bag retrieval proxy
  - ManiSkill bridge cloth retrieval proxy as the folded-cloth retrieval proxy
- the public benchmark package code and summaries
- the train/eval logs, checkpoints, cached datasets, validation sweeps, and correction logs for those runs
- full visual rerenders of the final `smoke_v5_eval_tuned_softerpref` dense-occlusion benchmark for both `trunk_only_ft` and `adapter_active_ft`
- the same-machine environment snapshot for the public benchmark stack used on this RunPod

## Top-Level Contents

- `code/`
  - the cleaned code snapshot inherited from `VLAarchtests3`
- `artifacts/`
  - prior staged checkpoints, proxy data, reports, and generated configs already bundled by `VLAarchtests3`
- `docs/`
  - prior handoff/audit docs plus the current public benchmark run logs and correction notes
- `legacy/`
  - older exact artifacts preserved by `VLAarchtests3`
- `setup/`
  - prior environment files plus a new public benchmark environment snapshot under `setup/public_benchmark/`
- `history/`
  - copied README history for `VLAarchtests`, `VLAarchtests2`, and `VLAarchtests3`
- `reports/`
  - the full current `/workspace/workspace/reports` tree from this machine
- `outputs/`
  - the full current `/workspace/workspace/outputs` tree from this machine
- `data/`
  - the full current `/workspace/workspace/data` tree from this machine
- `PUBLIC_BENCHMARK_RESULTS.md`
  - compact index of all public benchmark train/eval results from this session
- `MODEL_AND_ARTIFACT_INDEX.md`
  - practical map of the main artifact roots to start from

## Benchmark GIF Renders

The repo now also includes a full rendered replay of the final dense-occlusion benchmark:

- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref_gifs/`
  - `50` held-out `trunk_only_ft` gifs
  - `50` held-out `adapter_active_ft` gifs
  - `index.html`, `INDEX.md`, and `manifest.json` for browsing and validation
- renderer:
  - `code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/render_maniskill_pickclutter_benchmark_gifs.py`

Important caveats:

- these gifs are rerendered from the saved `smoke_v5_eval_tuned_softerpref` checkpoints and exact held-out seeds, not a different benchmark run
- the rerender kept the same `softer_pref` planner override used in the reported held-out result
- the rerender manifest records `0` success mismatches versus the saved benchmark json files
- only the dense-occlusion track has this full gif export right now

## Architecture State Carried Forward

The core model family inherited from `VLAarchtests3` is still:

- `trunk_only`
- `adapter_noop`
- `adapter_active`

The important architectural state carried into the public benchmark work is:

- wrapped-policy interface with exact `trunk_only`, `adapter_noop`, and `adapter_active` modes
- structured reveal/retrieve adapter with:
  - state prediction
  - task-routed proposal families
  - retrieve-feasibility gating
  - lightweight transition model
  - planner/reranker
- planner fixes that replaced hard vetoes with softer stage penalties in:
  - `code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/models/planner.py`

## Public Benchmark Summary

Detailed per-run results are in `PUBLIC_BENCHMARK_RESULTS.md`. The short version is:

### 1. Dense occluded retrieval proxy

Benchmark:

- ManiSkill `PickClutterYCB-v1`

Best current held-out result:

- directory:
  - `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref/`
- summary:
  - `trunk_only_ft = 0.04`
  - `adapter_noop = 0.04`
  - `adapter_active_ft = 0.62`
  - `delta_active_vs_trunk = +0.58`
  - `95% CI = [0.44, 0.72]`
  - `intervention_rate = 1.0`
  - `non_base_selection_rate = 1.0`

Important caveat:

- this was not a new retrain after `smoke_v5`
- it used the same `smoke_v5` checkpoints with planner hyperparameters selected on the frozen validation split and then applied once to the untouched held-out split

### 2. Bag retrieval proxy

Benchmark:

- public ManiSkill bridge basket retrieval proxy

Current fair read:

- seed `17` corrected held-out:
  - `trunk = 0.32`
  - `noop = 0.00`
  - `active = 0.48`
- seed `23` corrected held-out:
  - `trunk = 0.48`
  - `noop = 0.08`
  - `active = 0.48`
- corrected 2-seed aggregate:
  - `trunk = 0.40`
  - `noop = 0.04`
  - `active = 0.48`
  - `delta = +0.08`

Interpretation:

- bag remains modestly positive after using one consistent corrected planner across seeds
- the effect is smaller and less clean than the best occlusion result

### 3. Cloth retrieval proxy

Benchmark:

- public ManiSkill bridge cloth retrieval proxy

Current read:

- seed `17`:
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.10`
- seed `23`:
  - `trunk = 0.04`
  - `noop = 0.02`
  - `active = 0.02`
- seed `29`:
  - `trunk = 0.04`
  - `noop = 0.04`
  - `active = 0.04`
- 3-seed aggregate:
  - `trunk = 0.0400`
  - `noop = 0.0333`
  - `active = 0.0533`
  - `delta = +0.0133`

Interpretation:

- cloth is weak and unstable
- current evidence does not support a strong cloth-specific win

## Important Fairness Notes

The fairness story is mixed and should be stated plainly.

What is fair in the strongest public benchmark result:

- same initialization checkpoint for `trunk_only_ft` and `adapter_active_ft`
- same train/val/test split within each task
- same optimizer, LR, batch size, and unfreeze scope within each task
- `adapter_noop` is evaluated from the same adapter checkpoint as `adapter_active_ft`
- the held-out test episodes were not hand-picked after seeing outcomes

What is not fully paper-clean yet:

- most current public benchmark evidence is smoke-scale and low-seed
- the occlusion headline result depends on validation-selected planner tuning on top of a fixed checkpoint
- bag required eval-side planner correction for one seed to avoid a collapse
- cloth remains weak even after additional seeds and val sweeps

### PickClutter Split Fairness

The important point for the dense-occlusion track is that the dataset split did not drift across the early smoke versions.

- `data/maniskill_pickclutter/smoke_v1/episode_splits.json`
- `data/maniskill_pickclutter/smoke_v2/episode_splits.json`
- `data/maniskill_pickclutter/smoke_v3/episode_splits.json`

These files contain the same episode ids:

- train: `170000..170031`
- val: `171000..171007`
- eval: `172000..172049`

Also:

- there is no `data/maniskill_pickclutter/smoke_v4/`
- there is no `data/maniskill_pickclutter/smoke_v5/`

`smoke_v4` and `smoke_v5` were code/report version labels, not new held-out episode bundles.

### What Changed Across PickClutter Versions

The big changes across `smoke_v2`, `smoke_v3`, `smoke_v4`, and `smoke_v5` were:

- more benchmark-derived state supervision
- transition-model training enablement
- planner bug fixes
- fairness fixes so the adapter checkpoint did not hide a stronger shared trunk
- then a frozen-validation planner sweep for the final held-out eval

The big occlusion win was not caused by changing the eval episodes.

### Dense-Occlusion Render Artifacts

The final dense-occlusion run also has a full visual export in:

- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref_gifs/`

Those gifs show the robot interacting with the 3D scene and overlay the adaptor state per frame. For `adapter_active_ft`, the overlay includes:

- adaptor on/off state
- whether a non-base proposal was selected
- candidate index
- planner name
- planner score/confidence
- state signals such as visibility, access, gap, and damage

## Crucial Caveats

### Occlusion result was planner-tuned

The large jump in:

- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref/`

came from validation-selected planner tuning on top of the same `smoke_v5` checkpoint.

The selected override values were:

- `mode_preference_bonus = 0.75`
- `premature_retrieve_penalty = 0.5`
- `premature_insert_penalty = 0.25`
- `premature_maintain_penalty = 1.0`
- `occlusion_maintain_gap_min_access = 0.30`
- `occlusion_maintain_gap_min_visibility = 0.20`
- `retrieve_stage_access_threshold = 0.18`
- `retrieve_stage_reveal_threshold = 0.18`
- `retrieve_stage_support_threshold = 0.18`

That was a validation-only selection step. It was not a fresh retrain.

### Bag and cloth did not use real depth

The bridge-task runner for the bag and cloth proxies used:

- one real RGB camera
- copied into all camera slots
- zero-filled depth channels

The runner labels this stack:

- `rgb_triplicate_zero_depth`

This is a real limitation and it should not be hidden.

It happened because the bridge proxy runner used a compatibility shim to satisfy the shared multi-camera tensor interface without plumbing real bridge-scene multiview depth through the stack.

Consequences:

- bag and cloth are not modality-matched to the PickClutter runs
- PickClutter used real `rgbd_3cam`
- bag and cloth used weaker perception input

### Bag and cloth also used a different control wrapper

PickClutter:

- observation stack: `rgbd_3cam`
- action space: `bimanual_delta_pose`

Bag and cloth:

- observation stack: `rgb_triplicate_zero_depth`
- action space: `widowx_delta_pose`

So the cross-track story is architecture-consistent but not fully input/control-identical.

### `smoke_v4_evalprobe_fromv3` is not a clean retrain result

This run:

- `reports/maniskill_pickclutter_smoke_v4_evalprobe_fromv3/`

used corrected planner logic on top of `smoke_v3` weights. It is useful evidence that the active adapter can matter, but it is not a clean end-to-end retrain.

## What Was Actually Learned

The current repo supports the following claims:

- the structured adapter is still alive
- the active branch can clearly matter on a real public dense-occlusion benchmark proxy
- `adapter_noop` remains a useful fairness control
- bag-like retrieval still shows modest positive evidence
- cloth-like retrieval is currently the weak link

It does not support the following stronger claims yet:

- broad superiority on realistic manipulation benchmarks
- stable multi-seed wins across all three target-like public proxy tracks
- a clean modality-matched comparison across occlusion, bag, and cloth

## Environment And Setup

Two environment stories exist in this repo.

### Prior `VLAarchtests3` / RLBench stack

Preserved under:

- `setup/ENVIRONMENT.md`
- `setup/env_vars.sh`
- `setup/rlbench_pip_freeze.txt`

This is the older RLBench / AnyBimanual oriented environment.

### Current public benchmark stack

Preserved under:

- `setup/public_benchmark/ENVIRONMENT.md`
- `setup/public_benchmark/env_vars.sh`
- `setup/public_benchmark/python_version.txt`
- `setup/public_benchmark/uname.txt`
- `setup/public_benchmark/nvidia_smi.txt`
- `setup/public_benchmark/gpu_short.txt`
- `setup/public_benchmark/pip_freeze_python311.txt`
- `setup/public_benchmark/rlbench_env_pip_freeze.txt`
- `setup/public_benchmark/hf_env.txt`

The public benchmark runs in this session were assembled on:

- GPU: `NVIDIA L40S`
- VRAM: `46068 MiB`
- driver: `580.126.09`
- Python: `3.11.10`
- kernel: `Linux 6.8.0-88-generic`

## Recommended Starting Points

If you want the strongest current public benchmark evidence, start here:

- `docs/maniskill_pickclutter_correction_log_2026-04-01.md`
- `reports/maniskill_pickclutter_smoke_v5_eval_tuned_softerpref/public_benchmark_package_summary.json`

If you want the bag/cloth public bridge follow-up, start here:

- `docs/public_bridge_smoke_run_log_2026-04-01.md`
- `reports/maniskill_bag_bridge_eval_less_bonus_2seed_manual_summary.json`
- `reports/maniskill_cloth_bridge_val_sweep_seed23/summary.json`

If you want the repo lineage context, start here:

- `history/VLAarchtests_previous_README.md`
- `history/VLAarchtests2_previous_README.md`
- `history/VLAarchtests3_previous_README.md`

## Bottom Line

This repo is the complete organization package for the current workspace state.

It includes:

- the `VLAarchtests3` export base
- the full current machine `reports/`, `outputs/`, and `data/` trees
- the public benchmark code, datasets, checkpoints, and results
- the environment files needed to stand up the same stack on similar hardware

Use it as the archival handoff state for continuing the elastic-occlusion adapter work.

Do not cite it as if all three target-like public proxy tracks are already cleanly solved. The occlusion track is the strongest current evidence; bag is modest; cloth remains weak; and the bridge-task perception stack still needs a proper real-depth rewrite.