File size: 2,394 Bytes
2e0937e a1fc554 2e0937e a1fc554 2e0937e a1fc554 2e0937e a1fc554 2e0937e a1fc554 2e0937e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # VLAarchTests Bench
This repository is a bundled benchmark workspace for future VLAarchTests model
iterations, with emphasis on RLBench2 `take tray out of oven`.
## Included
- VLAarchTests benchmark code and generated public benchmark manifest
- Patched AnyBimanual RLBench runtime used to execute the public oven benchmark
- Patched PointFlowMatch + `diffusion_policy` source snapshot used to execute `take shoes out of box`
- Official `katefgroup/3d_flowmatch_actor` PerAct2 checkpoint and public test data
- Official PointFlowMatch `1717447341-indigo-quokka` checkpoint for `take_shoes_out_of_box`
- Public AnyBimanual LF baseline weights and comparison logs
- Verified benchmark reports:
- oven subset run: `9/10`
- oven full official run: `95/100 = 0.95`
- shoes GPU search: non-zero success verified before later simulator crash
- hybrid public benchmark smoke outputs
- DexGarmentLab benchmark-related validation scripts and validation logs
## Key Result
The strongest public out-of-box checkpoint validated here is:
- `models/3dfa_peract2/3dfa_peract2.pth`
Official oven result artifacts:
- `reports/3dfa_peract2_take_tray_out_of_oven_subset10/eval_after_official_ttm.json`
- `reports/3dfa_peract2_take_tray_out_of_oven_full100/eval.json`
Shoes result artifacts:
- `reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/summary.json`
- `reports/pointflowmatch_take_shoes_out_of_box_ep10_k50_gpu/run.log`
## Important Code Paths
- `code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/public_benchmark_package.py`
- `code/VLAarchtests4/code/VLAarchtests2_code/VLAarchtests/code/reveal_vla_bimanual/eval/run_rlbench_hybrid_smoke.py`
- `third_party/AnyBimanual/third_party/RLBench/rlbench/bimanual_tasks/bimanual_take_tray_out_of_oven.py`
- `third_party/AnyBimanual/third_party/RLBench/rlbench/task_ttms/bimanual_take_tray_out_of_oven.ttm`
- `third_party/PointFlowMatch/pfp/envs/rlbench_env.py`
- `third_party/PointFlowMatch/pfp/policy/fm_policy.py`
- `scripts/run_pointflowmatch_take_shoes_out_of_box.sh`
## External Dependencies Not Mirrored Here
- CoppeliaSim v4.1.0 binary runtime
- Local Python environments under `/workspace/envs`
- Full IsaacSim installation
- Full DexGarmentLab simulator assets beyond the benchmark-related scripts and logs
See `docs/ENVIRONMENT_NOTES.md` for the runtime notes used in this workspace.
|