README: portable setup (no absolute paths), capability table, agentic verification
Browse files
README.md
CHANGED
|
@@ -117,47 +117,81 @@ source/bimanual/yam/
|
|
| 117 |
scripts/ runners, asset converters, verification
|
| 118 |
```
|
| 119 |
|
| 120 |
-
##
|
| 121 |
|
| 122 |
-
**
|
| 123 |
-
|
| 124 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
|
| 126 |
```bash
|
| 127 |
-
|
|
|
|
|
|
|
| 128 |
```
|
| 129 |
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
|
|
|
| 136 |
|
| 137 |
```bash
|
| 138 |
-
export ROBOTWIN_USD=/
|
|
|
|
| 139 |
export OMNI_KIT_ACCEPT_EULA=YES
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
|
|
|
|
|
|
|
|
|
| 141 |
python scripts/yam_task.py --list
|
| 142 |
python scripts/yam_task.py --task grape_box --seed 3 \
|
| 143 |
--video outputs/tasks/grape_box.mp4 \
|
| 144 |
--kit_args="--/rtx/verifyDriverVersion/enabled=false"
|
| 145 |
```
|
| 146 |
|
| 147 |
-
Two
|
| 148 |
|
| 149 |
* `OMNI_KIT_ACCEPT_EULA=YES` — otherwise Omniverse blocks on an interactive EULA prompt and the
|
| 150 |
-
run hangs forever with no output.
|
| 151 |
-
* `--kit_args="--/rtx/verifyDriverVersion/enabled=false"` —
|
| 152 |
-
`535.309.01`
|
| 153 |
-
wrong, not the driver.
|
| 154 |
|
| 155 |
-
Extras: `--no-randomize` (nominal poses), `--set gripper_effort=70`
|
| 156 |
-
`--first-frame out.png` (build the scene, save one frame, exit
|
|
|
|
| 157 |
|
| 158 |
-
**Config precedence:** `class attribute → tasks/configs/<task>.yaml → --set`.
|
| 159 |
-
YAML mirroring its parameters *with every key commented out on purpose* — an
|
| 160 |
-
overrides the Python and keeps overriding it, so a later code change silently
|
|
|
|
| 161 |
|
| 162 |
Full architecture, task-authoring guide, the asset-gotcha table and the verification workflow:
|
| 163 |
[`source/bimanual/yam/README.md`](source/bimanual/yam/README.md).
|
|
@@ -199,6 +233,30 @@ around, so several tasks moved from green to red on purpose.
|
|
| 199 |
|
| 200 |
Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`.
|
| 201 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
<!-- STATUS TABLE -->
|
| 203 |
**18 of 30 passing.**
|
| 204 |
|
|
|
|
| 117 |
scripts/ runners, asset converters, verification
|
| 118 |
```
|
| 119 |
|
| 120 |
+
## What this suite supports
|
| 121 |
|
| 122 |
+
**30 scripted bimanual manipulation tasks** on the YAM arm in Isaac Lab, built on RoboTwin 2.0
|
| 123 |
+
assets. Not a policy — every task ships a scripted solver, so it produces demonstrations and a
|
| 124 |
+
pass/fail signal you can trust.
|
| 125 |
+
|
| 126 |
+
| capability | what it means |
|
| 127 |
+
|---|---|
|
| 128 |
+
| **Skill families** | pick-and-place · multi-object · stacking · sorting · insertion · pushing (single and relay) · bimanual lift · handover · tool use (sweep, wipe) · pouring · constrained pull · articulated fixtures · deformable chain |
|
| 129 |
+
| **Both arms** | either arm alone, or both on one motion profile (`move_both`) — driving them in sequence parks the first in the second's path |
|
| 130 |
+
| **Randomization** | every object, fixture, container and marker takes `xy_jitter` / `yaw_jitter`; `--seed` reproduces an episode exactly. 52 of 56 spawn calls are randomized |
|
| 131 |
+
| **Articulated props** | SAPIEN mobility URDFs — hinged doors, sliding drawers, lids, toggles. Joint values are first-class state and can be success conditions |
|
| 132 |
+
| **Chains / rope** | a rope as a jointed chain, judged on its own geometry (end-to-end span) rather than on a placement |
|
| 133 |
+
| **Configuration** | class attribute → per-task YAML → `--set key=value`, resolved *before* the simulator starts |
|
| 134 |
+
| **Asset pipeline** | rigid → USD, hollow containers, articulated URDFs, plus procedurally generated meshes (ring, rope, pot lid) |
|
| 135 |
+
| **Agentic verification** | rendered contact sheets + review prompts for a vision agent, verdicts fed back into concrete parameter fixes, and task proposal from the asset library — `scripts/yam_agent_loop.py` |
|
| 136 |
+
| **Verification** | strict numeric conditions **plus** the visual check above; `scripts/status.py` regenerates the status table from run logs |
|
| 137 |
+
| **Outputs** | per-episode MP4 with an action/state HUD, contact sheets, single-frame scene renders |
|
| 138 |
+
|
| 139 |
+
## Environment
|
| 140 |
+
|
| 141 |
+
Isaac Sim 5.0 + Isaac Lab 2.2.0, Python 3.11, an NVIDIA GPU (the PhysX GPU pipeline is required).
|
| 142 |
+
|
| 143 |
+
Isaac Lab may already exist on the machine inside another project's virtualenv. If so, prefer an
|
| 144 |
+
**overlay venv** — a fresh venv whose `.pth` file points at that interpreter's `site-packages` —
|
| 145 |
+
over reinstalling Isaac Sim:
|
| 146 |
|
| 147 |
```bash
|
| 148 |
+
python -m venv .venv
|
| 149 |
+
echo "/path/to/existing/isaac/venv/lib/python3.11/site-packages" \
|
| 150 |
+
> .venv/lib/python3.11/site-packages/_isaac_overlay.pth
|
| 151 |
```
|
| 152 |
|
| 153 |
+
Isaac Lab's *source checkout* is also needed for `isaaclab_tasks` / `isaaclab_assets`; add it the
|
| 154 |
+
same way if it is not pip-installed.
|
| 155 |
+
|
| 156 |
+
### Configure
|
| 157 |
+
|
| 158 |
+
Everything is driven by two environment variables and a relative path. Nothing below hardcodes a
|
| 159 |
+
machine-specific location:
|
| 160 |
|
| 161 |
```bash
|
| 162 |
+
export ROBOTWIN_USD=./robotwin_usd # converted USD assets (see Assets)
|
| 163 |
+
export ROBOTWIN_SRC=./robotwin_object/objects # source meshes, only needed to convert
|
| 164 |
export OMNI_KIT_ACCEPT_EULA=YES
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
`ROBOTWIN_USD` defaults are read at import time, so export it before running anything. Unzip
|
| 168 |
+
`assets/robotwin_usd.zip` from this repo's HF page to populate it.
|
| 169 |
|
| 170 |
+
### Run
|
| 171 |
+
|
| 172 |
+
```bash
|
| 173 |
python scripts/yam_task.py --list
|
| 174 |
python scripts/yam_task.py --task grape_box --seed 3 \
|
| 175 |
--video outputs/tasks/grape_box.mp4 \
|
| 176 |
--kit_args="--/rtx/verifyDriverVersion/enabled=false"
|
| 177 |
```
|
| 178 |
|
| 179 |
+
Two flags are commonly required and easy to miss:
|
| 180 |
|
| 181 |
* `OMNI_KIT_ACCEPT_EULA=YES` — otherwise Omniverse blocks on an interactive EULA prompt and the
|
| 182 |
+
run hangs forever with no output at all.
|
| 183 |
+
* `--kit_args="--/rtx/verifyDriverVersion/enabled=false"` — needed on drivers whose version
|
| 184 |
+
string Omniverse misparses (e.g. `535.309.01` read as `535.53` and rejected as too old). The
|
| 185 |
+
check is wrong, not the driver. Drop this flag if your driver passes the check.
|
| 186 |
|
| 187 |
+
Extras: `--no-randomize` (nominal poses, for reproducing a failure), `--set gripper_effort=70`
|
| 188 |
+
(override any task parameter), `--first-frame out.png` (build the scene, save one frame, exit
|
| 189 |
+
without solving).
|
| 190 |
|
| 191 |
+
**Config precedence:** `class attribute → source/bimanual/yam/tasks/configs/<task>.yaml → --set`.
|
| 192 |
+
Each task ships a YAML mirroring its parameters *with every key commented out on purpose* — an
|
| 193 |
+
active key there overrides the Python and keeps overriding it, so a later code change silently
|
| 194 |
+
does nothing.
|
| 195 |
|
| 196 |
Full architecture, task-authoring guide, the asset-gotcha table and the verification workflow:
|
| 197 |
[`source/bimanual/yam/README.md`](source/bimanual/yam/README.md).
|
|
|
|
| 233 |
|
| 234 |
Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`.
|
| 235 |
|
| 236 |
+
### Agent-in-the-loop verification
|
| 237 |
+
|
| 238 |
+
The numeric check only inspects final object poses. It will happily pass an episode where the cup
|
| 239 |
+
landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target
|
| 240 |
+
without ever tipping — all three of those actually happened here. So every task is also reviewed
|
| 241 |
+
visually, and only `numeric=SUCCESS` **and** `visual=CONFIRMED` counts as done.
|
| 242 |
+
|
| 243 |
+
```bash
|
| 244 |
+
python scripts/contact_sheet.py outputs/tasks/*.mp4 # episode -> one strip of frames
|
| 245 |
+
python scripts/yam_agent_loop.py evaluate # contact sheets + per-task review prompts
|
| 246 |
+
python scripts/yam_agent_loop.py update # verdicts -> concrete parameter fixes
|
| 247 |
+
python scripts/yam_agent_loop.py propose # uncovered skill axes from the asset library
|
| 248 |
+
```
|
| 249 |
+
|
| 250 |
+
`evaluate` renders a sheet per episode and emits a prompt naming what to look for in that
|
| 251 |
+
specific task (is the cup upright? is anything actually in the gripper? did the lid come off
|
| 252 |
+
*before* the food went in?). `update` maps a verdict onto a fix through a symptom→cause table
|
| 253 |
+
built from the failures in [`DIAGNOSTICS.md`](DIAGNOSTICS.md) — e.g. *"jaws closed on nothing"* →
|
| 254 |
+
measure the mesh origin offset, not the clamp force. `propose` reads the converted asset library
|
| 255 |
+
and suggests skills not yet covered.
|
| 256 |
+
|
| 257 |
+
Reviewing this way is what caught the three false positives described above; the fix in each case
|
| 258 |
+
was to tighten the numeric condition, so the visual pass and the numeric pass now agree.
|
| 259 |
+
|
| 260 |
<!-- STATUS TABLE -->
|
| 261 |
**18 of 30 passing.**
|
| 262 |
|