Add grab_cubes_skin_3cam and lift_leave_cubes_skin_3cam; explicit training prompts

README.md

@@ -2,7 +2,7 @@
 library_name: openpi
 license: apache-2.0
 tags: [robotics, pi0, pi05, openpi, so101, lerobot, tactile]
-datasets: [endoard/grab_ball_3cam_skin, endoard/grab_cup_3cam_skin, endoard/grab_red_cube_3cam_skin, endoard/grab_green_cube_3cam_skin, endoard/grab_green_cylinder_3cam_skin]
+datasets: [endoard/grab_ball_3cam_skin, endoard/grab_cup_3cam_skin, endoard/grab_red_cube_3cam_skin, endoard/grab_green_cube_3cam_skin, endoard/grab_green_cylinder_3cam_skin, endoard/grab_red_cube_distractor_green_cube_3cam_skin, endoard/grab_green_cube_distractor_red_cube_3cam_skin]
 ---
 
 # pi0.5 SO-101 with a tactile skin
@@ -19,11 +19,42 @@ None has been evaluated on the robot yet.
 | `grab_ball_skin_3cam/` | `pi05_so101_grab_ball_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the ball" | `endoard/grab_ball_3cam_skin`, 150 episodes / 65,989 frames | 10000, 20000, 29999 |
 | `grab_cup_skin_3cam/` | `pi05_so101_grab_cup_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the blue cup" | `endoard/grab_cup_3cam_skin`, 166 episodes / 64,812 frames | 10000, 20000, 29999 |
 | `grab_3obj_skin_3cam/` | `pi05_so101_grab_3obj_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" / "Grab the green cylinder" | `endoard/grab_{red_cube,green_cube,green_cylinder}_3cam_skin` merged, 450 episodes / 161,918 frames, one task per source | 10000, 20000, 28000 |
+| `grab_cubes_skin_3cam/` | `pi05_so101_grab_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" (see [Prompts](#prompts)) | `endoard/grab_{red_cube,green_cube,red_cube_distractor_green_cube,green_cube_distractor_red_cube}_3cam_skin` merged, 550 episodes / 191,324 frames | 10000 (20000, 29999 in training) |
+| `lift_leave_cubes_skin_3cam/` | `pi05_so101_lift_leave_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Lift the red cube" / "Lift the green cube" / "Lift the red cube, leave the green cube on the table" / "Lift the green cube, leave the red cube on the table" (see [Prompts](#prompts)) | same 4 cube datasets merged, 550 episodes / 191,324 frames | 10000, 20000 (29999 in training) |
 
 `grab_3obj_skin_3cam` is multi-task: the prompt selects the object. Its run was stopped at
 step 28,500 of 30k, so 28000 is its last checkpoint. Its held-out split is stratified per task
 (15 episodes each).
 
+`grab_cubes_skin_3cam` and `lift_leave_cubes_skin_3cam` are trained on the same 550 cube episodes
+with different prompts; their held-out split is stratified per source (15/15/15/10 episodes).
+
+## Prompts
+
+Every policy here was trained on a fixed set of text prompts. At inference send one of them
+**verbatim** as the observation `prompt` (same wording, case and punctuation); other phrasings
+are untested.
+
+| folder | training prompts (exact strings) |
+|---|---|
+| `grab_ball_skin/`, `grab_ball_skin_3cam/` | `Grab the ball` |
+| `grab_cup_skin_3cam/` | `Grab the blue cup` |
+| `grab_3obj_skin_3cam/` | `Grab the red cube`, `Grab the green cube`, `Grab the green cylinder` |
+| `grab_cubes_skin_3cam/` | `Grab the red cube`, `Grab the green cube` |
+| `lift_leave_cubes_skin_3cam/` | `Lift the red cube`, `Lift the green cube`, `Lift the red cube, leave the green cube on the table`, `Lift the green cube, leave the red cube on the table` |
+
+Which prompt each source dataset was trained with in the two cube runs:
+
+| source dataset (episodes) | scene | `grab_cubes_skin_3cam` | `lift_leave_cubes_skin_3cam` |
+|---|---|---|---|
+| `endoard/grab_red_cube_3cam_skin` (150) | red cube alone | `Grab the red cube` | `Lift the red cube` |
+| `endoard/grab_green_cube_3cam_skin` (150) | green cube alone | `Grab the green cube` | `Lift the green cube` |
+| `endoard/grab_red_cube_distractor_green_cube_3cam_skin` (150) | red cube + green cube | `Grab the red cube` | `Lift the red cube, leave the green cube on the table` |
+| `endoard/grab_green_cube_distractor_red_cube_3cam_skin` (100) | green cube + red cube | `Grab the green cube` | `Lift the green cube, leave the red cube on the table` |
+
+So `grab_cubes_skin_3cam` uses one prompt per target colour whether or not the other cube is
+on the table, and `lift_leave_cubes_skin_3cam` also names the distractor when there is one.
+
 Each step folder is an openpi checkpoint without optimizer state: `params/` (EMA weights),
 `assets/<asset_id>/norm_stats.json` (quantile stats), `_CHECKPOINT_METADATA`, and a `README.md`
 with that checkpoint's details.
@@ -57,6 +88,7 @@ The 3-camera policies need all three cameras every step.
 - Action 6-D (padded to 32): **delta on the 5 joints, absolute gripper** (`make_bool_mask(5, -1)`).
 - Images: `base_0_rgb` = `fixed`, `left_wrist_0_rgb` = `handeye`; `right_wrist_0_rgb` = `side`
   (mask on) for 3-camera runs, zeros with mask off for `grab_ball_skin`.
-- `action_horizon=16`, `action_stride=2`, prompt from the dataset task string.
+- `action_horizon=16`, `action_stride=2`, prompt from the dataset task string (remapped per
+  source for the two cube runs, see [Prompts](#prompts)).
 - From `pi05_base`, 4 nodes x 1 A100-64GB, `--fsdp-devices=4`, batch 32, AdamW with cosine
   decay, EMA 0.99, seed 42, 30k steps, 10% of episodes held out.