yqi19 commited on
Commit
9b8bfbc
·
verified ·
1 Parent(s): 5774ceb

README: portable setup (no absolute paths), capability table, agentic verification

Browse files
Files changed (1) hide show
  1. README.md +80 -22
README.md CHANGED
@@ -117,47 +117,81 @@ source/bimanual/yam/
117
  scripts/ runners, asset converters, verification
118
  ```
119
 
120
- ## Environment (this machine)
121
 
122
- **There is no `env_isaaclab` conda env here.** Isaac Sim 5.0 / Isaac Lab 2.2.0 live inside
123
- `/home/yu/robolab_project/RoboLab/.venv`, so this project uses an *overlay venv* that adds that
124
- interpreter's `site-packages` through a `.pth` file rather than reinstalling anything:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
125
 
126
  ```bash
127
- /home/yu/internship_yu/yam_venv/bin/python # use this interpreter for everything below
 
 
128
  ```
129
 
130
- | Component | Version | Location |
131
- |---|---|---|
132
- | Isaac Sim | 5.0 | bundled in the RoboLab venv |
133
- | Isaac Lab | 2.2.0 | `/home/yu/internship_yu/IsaacLab_v2.2.0` (source checkout, for `isaaclab_tasks` / `isaaclab_assets`) |
134
- | Python | 3.11 | RoboLab venv |
135
- | GPU | NVIDIA + CUDA | required — PhysX GPU pipeline |
 
136
 
137
  ```bash
138
- export ROBOTWIN_USD=/home/yu/internship_yu/robotwin_usd
 
139
  export OMNI_KIT_ACCEPT_EULA=YES
 
 
 
 
140
 
 
 
 
141
  python scripts/yam_task.py --list
142
  python scripts/yam_task.py --task grape_box --seed 3 \
143
  --video outputs/tasks/grape_box.mp4 \
144
  --kit_args="--/rtx/verifyDriverVersion/enabled=false"
145
  ```
146
 
147
- Two of those are **not optional on this machine**:
148
 
149
  * `OMNI_KIT_ACCEPT_EULA=YES` — otherwise Omniverse blocks on an interactive EULA prompt and the
150
- run hangs forever with no output.
151
- * `--kit_args="--/rtx/verifyDriverVersion/enabled=false"` — the installed driver reports
152
- `535.309.01`, which Omniverse's parser reads as `535.53` and rejects as too old. The check is
153
- wrong, not the driver.
154
 
155
- Extras: `--no-randomize` (nominal poses), `--set gripper_effort=70` (override any parameter),
156
- `--first-frame out.png` (build the scene, save one frame, exit without solving).
 
157
 
158
- **Config precedence:** `class attribute → tasks/configs/<task>.yaml → --set`. Each task ships a
159
- YAML mirroring its parameters *with every key commented out on purpose* — an active key there
160
- overrides the Python and keeps overriding it, so a later code change silently does nothing.
 
161
 
162
  Full architecture, task-authoring guide, the asset-gotcha table and the verification workflow:
163
  [`source/bimanual/yam/README.md`](source/bimanual/yam/README.md).
@@ -199,6 +233,30 @@ around, so several tasks moved from green to red on purpose.
199
 
200
  Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`.
201
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
  <!-- STATUS TABLE -->
203
  **18 of 30 passing.**
204
 
 
117
  scripts/ runners, asset converters, verification
118
  ```
119
 
120
+ ## What this suite supports
121
 
122
+ **30 scripted bimanual manipulation tasks** on the YAM arm in Isaac Lab, built on RoboTwin 2.0
123
+ assets. Not a policy — every task ships a scripted solver, so it produces demonstrations and a
124
+ pass/fail signal you can trust.
125
+
126
+ | capability | what it means |
127
+ |---|---|
128
+ | **Skill families** | pick-and-place · multi-object · stacking · sorting · insertion · pushing (single and relay) · bimanual lift · handover · tool use (sweep, wipe) · pouring · constrained pull · articulated fixtures · deformable chain |
129
+ | **Both arms** | either arm alone, or both on one motion profile (`move_both`) — driving them in sequence parks the first in the second's path |
130
+ | **Randomization** | every object, fixture, container and marker takes `xy_jitter` / `yaw_jitter`; `--seed` reproduces an episode exactly. 52 of 56 spawn calls are randomized |
131
+ | **Articulated props** | SAPIEN mobility URDFs — hinged doors, sliding drawers, lids, toggles. Joint values are first-class state and can be success conditions |
132
+ | **Chains / rope** | a rope as a jointed chain, judged on its own geometry (end-to-end span) rather than on a placement |
133
+ | **Configuration** | class attribute → per-task YAML → `--set key=value`, resolved *before* the simulator starts |
134
+ | **Asset pipeline** | rigid → USD, hollow containers, articulated URDFs, plus procedurally generated meshes (ring, rope, pot lid) |
135
+ | **Agentic verification** | rendered contact sheets + review prompts for a vision agent, verdicts fed back into concrete parameter fixes, and task proposal from the asset library — `scripts/yam_agent_loop.py` |
136
+ | **Verification** | strict numeric conditions **plus** the visual check above; `scripts/status.py` regenerates the status table from run logs |
137
+ | **Outputs** | per-episode MP4 with an action/state HUD, contact sheets, single-frame scene renders |
138
+
139
+ ## Environment
140
+
141
+ Isaac Sim 5.0 + Isaac Lab 2.2.0, Python 3.11, an NVIDIA GPU (the PhysX GPU pipeline is required).
142
+
143
+ Isaac Lab may already exist on the machine inside another project's virtualenv. If so, prefer an
144
+ **overlay venv** — a fresh venv whose `.pth` file points at that interpreter's `site-packages` —
145
+ over reinstalling Isaac Sim:
146
 
147
  ```bash
148
+ python -m venv .venv
149
+ echo "/path/to/existing/isaac/venv/lib/python3.11/site-packages" \
150
+ > .venv/lib/python3.11/site-packages/_isaac_overlay.pth
151
  ```
152
 
153
+ Isaac Lab's *source checkout* is also needed for `isaaclab_tasks` / `isaaclab_assets`; add it the
154
+ same way if it is not pip-installed.
155
+
156
+ ### Configure
157
+
158
+ Everything is driven by two environment variables and a relative path. Nothing below hardcodes a
159
+ machine-specific location:
160
 
161
  ```bash
162
+ export ROBOTWIN_USD=./robotwin_usd # converted USD assets (see Assets)
163
+ export ROBOTWIN_SRC=./robotwin_object/objects # source meshes, only needed to convert
164
  export OMNI_KIT_ACCEPT_EULA=YES
165
+ ```
166
+
167
+ `ROBOTWIN_USD` defaults are read at import time, so export it before running anything. Unzip
168
+ `assets/robotwin_usd.zip` from this repo's HF page to populate it.
169
 
170
+ ### Run
171
+
172
+ ```bash
173
  python scripts/yam_task.py --list
174
  python scripts/yam_task.py --task grape_box --seed 3 \
175
  --video outputs/tasks/grape_box.mp4 \
176
  --kit_args="--/rtx/verifyDriverVersion/enabled=false"
177
  ```
178
 
179
+ Two flags are commonly required and easy to miss:
180
 
181
  * `OMNI_KIT_ACCEPT_EULA=YES` — otherwise Omniverse blocks on an interactive EULA prompt and the
182
+ run hangs forever with no output at all.
183
+ * `--kit_args="--/rtx/verifyDriverVersion/enabled=false"` — needed on drivers whose version
184
+ string Omniverse misparses (e.g. `535.309.01` read as `535.53` and rejected as too old). The
185
+ check is wrong, not the driver. Drop this flag if your driver passes the check.
186
 
187
+ Extras: `--no-randomize` (nominal poses, for reproducing a failure), `--set gripper_effort=70`
188
+ (override any task parameter), `--first-frame out.png` (build the scene, save one frame, exit
189
+ without solving).
190
 
191
+ **Config precedence:** `class attribute → source/bimanual/yam/tasks/configs/<task>.yaml → --set`.
192
+ Each task ships a YAML mirroring its parameters *with every key commented out on purpose* — an
193
+ active key there overrides the Python and keeps overriding it, so a later code change silently
194
+ does nothing.
195
 
196
  Full architecture, task-authoring guide, the asset-gotcha table and the verification workflow:
197
  [`source/bimanual/yam/README.md`](source/bimanual/yam/README.md).
 
233
 
234
  Run `python scripts/status.py` to regenerate the table below from `outputs/tasks/*.log`.
235
 
236
+ ### Agent-in-the-loop verification
237
+
238
+ The numeric check only inspects final object poses. It will happily pass an episode where the cup
239
+ landed upside-down, a gripper hung empty in the air, or a vessel was carried over the target
240
+ without ever tipping — all three of those actually happened here. So every task is also reviewed
241
+ visually, and only `numeric=SUCCESS` **and** `visual=CONFIRMED` counts as done.
242
+
243
+ ```bash
244
+ python scripts/contact_sheet.py outputs/tasks/*.mp4 # episode -> one strip of frames
245
+ python scripts/yam_agent_loop.py evaluate # contact sheets + per-task review prompts
246
+ python scripts/yam_agent_loop.py update # verdicts -> concrete parameter fixes
247
+ python scripts/yam_agent_loop.py propose # uncovered skill axes from the asset library
248
+ ```
249
+
250
+ `evaluate` renders a sheet per episode and emits a prompt naming what to look for in that
251
+ specific task (is the cup upright? is anything actually in the gripper? did the lid come off
252
+ *before* the food went in?). `update` maps a verdict onto a fix through a symptom→cause table
253
+ built from the failures in [`DIAGNOSTICS.md`](DIAGNOSTICS.md) — e.g. *"jaws closed on nothing"* →
254
+ measure the mesh origin offset, not the clamp force. `propose` reads the converted asset library
255
+ and suggests skills not yet covered.
256
+
257
+ Reviewing this way is what caught the three false positives described above; the fix in each case
258
+ was to tighten the numeric condition, so the visual pass and the numeric pass now agree.
259
+
260
  <!-- STATUS TABLE -->
261
  **18 of 30 passing.**
262