| # VLAarchTests Bench |
|
|
| This repository is a bundled benchmark workspace for future VLAarchTests model |
| iterations, with emphasis on RLBench2 `take tray out of oven`. |
|
|
| ## Included |
|
|
| - VLAarchTests benchmark code and generated public benchmark manifest |
| - Patched AnyBimanual RLBench runtime used to execute the public oven benchmark |
| - Patched PointFlowMatch + `diffusion_policy` source snapshot used to execute `take shoes out of box` |
| - Official `katefgroup/3d_flowmatch_actor` PerAct2 checkpoint and public test data |
| - Official PointFlowMatch `1717447341-indigo-quokka` checkpoint for `take_shoes_out_of_box` |
| - Public AnyBimanual LF baseline weights and comparison logs |
| - Verified benchmark reports: |
| - oven subset run: `9/10` |
| - oven full official run: `95/100 = 0.95` |
| - shoes GPU search: non-zero success verified before later simulator crash |
| - hybrid public benchmark smoke outputs |
| - DexGarmentLab benchmark-related validation scripts and validation logs |
|
|
| ## Key Result |
|
|
| The strongest public out-of-box checkpoint validated here is: |
|
|
| - `models/3dfa_peract2/3dfa_peract2.pth` |
|
|
| Official oven result artifacts: |
|
|
| - `reports/3dfa_peract2_take_tray_out_of_oven_subset10/eval_after_official_ttm.json` |
| - `reports/3dfa_peract2_take_tray_out_of_oven_full100/eval.json` |
|
|
| Shoes result artifacts: |
|
|
| - `reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/summary.json` |
| - `reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/run.log` |
|
|
| ## Important Code Paths |
|
|
| - `code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/public_benchmark_package.py` |
| - `code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/run_rlbench_hybrid_smoke.py` |
| - `third_party/AnyBimanual/third_party/RLBench/rlbench/bimanual_tasks/bimanual_take_tray_out_of_oven.py` |
| - `third_party/AnyBimanual/third_party/RLBench/rlbench/task_ttms/bimanual_take_tray_out_of_oven.ttm` |
| - `third_party/PointFlowMatch/pfp/envs/rlbench_env.py` |
| - `third_party/PointFlowMatch/pfp/policy/fm_policy.py` |
| - `scripts/run_pointflowmatch_take_shoes_out_of_box.sh` |
|
|
| ## External Dependencies Not Mirrored Here |
|
|
| - CoppeliaSim v4.1.0 binary runtime |
| - Local Python environments under `/workspace/envs` |
| - Full IsaacSim installation |
| - Full DexGarmentLab simulator assets beyond the benchmark-related scripts and logs |
|
|
| See `docs/ENVIRONMENT_NOTES.md` for the runtime notes used in this workspace. |
|
|
