outputs/droid_coverage/bundle/README.md

# DROID coverage sample (56 episodes)

Chosen to **cover DROID's axes of variation**, not sampled uniformly. A uniform draw
returns mostly TRI-lab pick-and-place of rigid objects; this spreads over the
properties that change what a geometry pipeline has to do.

Only camera `ext1` is included. Every episode here is also covered by PointWorld's
camera set, so annotations line up if you go further.

## Two properties of DROID worth knowing before you look

1. **`current_task` is usually a category template, not a description.** Strings like
   `"Move object into or out of container (ex: drawer, clothes hamper, plate, trashcan, washer)"`
   or `"Do any task, and then reset the scene."` are the norm. Object identity has to
   come from the separate human-annotation file
   (`aggregated-annotations-030724.json`, 3 instructions per episode).
2. **The human annotations cover successful episodes only.** Measured over 1,153
   episodes' metadata: all 832 with annotations are successes, all 125 failures have
   none. So any object-based filter silently selects for success. The
   `unknown_no_annotation` rows below are failure episodes drawn deliberately, and
   their object identity is genuinely unknown.

## Object physics class

The axis that decides whether one rigid SE(3) pose per frame is even the right model.

| class | n |
|---|---|
| articulated | 7 |
| deformable | 7 |
| flat_thin | 7 |
| organic | 6 |
| rigid_compact | 7 |
| rigid_symmetric | 6 |
| rigid_thin | 6 |
| transparent | 2 |
| unknown_no_annotation | 8 |

## Task template

| class | n |
|---|---|
| clean_with_cloth | 5 |
| concrete | 8 |
| fold | 5 |
| free_form | 6 |
| hang | 4 |
| into_container | 7 |
| lid_on_off | 7 |
| open_close | 5 |
| press_button | 4 |
| reposition | 5 |

## Lab

| lab | n |
|---|---|
| AUTOLab | 6 |
| CLVR | 7 |
| GuptaLab | 6 |
| ILIAD | 6 |
| IPRL | 7 |
| IRIS | 8 |
| PennPAL | 4 |
| RAD | 4 |
| RAIL | 5 |
| REAL | 3 |

## Outcome

| success | n |
|---|---|
| False | 8 |
| True | 48 |

## Episode length (frames)

min 117 / median 307 / max 2449

## Files

- `index.csv` -- one row per episode
- `manifest.json` -- the picker's raw output, including all 3 human instructions
- `thumbs/<uuid>.jpg` -- 3-frame strip per episode (5% / 45% / 90%)
- `sheets/<object_class>.jpg` -- all episodes of one class stacked
- `videos/<uuid>.mp4` -- the ext1 clip

## Episodes

| uuid | object class | task | lab | frames | ok | instruction |
|---|---|---|---|---|---|---|
| `CLVR+13759f6e+2023-06-03-23h-29m-55s` | articulated | press_button | CLVR | 207 | y | Close the lid on the silver flask |
| `CLVR+13759f6e+2023-06-03-23h-27m-33s` | articulated | press_button | CLVR | 140 | y | Close the lid on the top of the thermos flask |
| `CLVR+13759f6e+2023-06-03-00h-10m-07s` | articulated | press_button | CLVR | 132 | y | Open the lid at the top of the thermos flask |
| `GuptaLab+553d1bd5+2023-05-28-15h-35m-41s` | articulated | open_close | GuptaLab | 227 | y | Open the bottom drawer of the cabinet to the right. |
| `PennPAL+06b0ffa5+2023-04-29-20h-25m-37s` | articulated | open_close | PennPAL | 117 | y | Close the drawer |
| `RAD+62828209+2023-09-09-07h-22m-33s` | articulated | press_button | RAD | 175 | y | Press the button on the microwave |
| `RAIL+80edfcb1+2023-06-29-12h-44m-09s` | articulated | lid_on_off | RAIL | 428 | y | Pick the lid and put it on the pot |
| `CLVR+236539bc+2023-05-29-21h-48m-38s` | deformable | hang | CLVR | 225 | y | Put the cloth on the stand |
| `ILIAD+50aee79f+2023-08-17-21h-00m-34s` | deformable | fold | ILIAD | 961 | y | Turn the towel over unto the left |
| `ILIAD+50aee79f+2023-08-17-21h-23m-04s` | deformable | fold | ILIAD | 637 | y | Unfold the grey towel. |
| `IRIS+226b9f35+2023-07-11-14h-00m-58s` | deformable | fold | IRIS | 600 | y | Fold the towel |
| `PennPAL+acda9df3+2023-07-07-20h-24m-56s` | deformable | hang | PennPAL | 254 | y | Unhang the light yellow cloth from the backrest of the chair |
| `PennPAL+acda9df3+2023-07-07-20h-09m-22s` | deformable | fold | PennPAL | 393 | y | Fold the thing on the table |
| `RAIL+80edfcb1+2023-04-26-15h-45m-39s` | deformable | clean_with_cloth | RAIL | 191 | y | Move the towel |
| `AUTOLab+5d05c5aa+2023-11-12-17h-00m-03s` | flat_thin | concrete | AUTOLab | 1468 | y | Pick up all the objects surrounding the white plate and arrange them o |
| `CLVR+13759f6e+2023-06-19-17h-10m-03s` | flat_thin | into_container | CLVR | 333 | y | Remove the objects from the tray |
| `ILIAD+5e938e3b+2023-07-20-10h-24m-39s` | flat_thin | open_close | ILIAD | 296 | y | Pull and push the pullout table |
| `IRIS+226b9f35+2023-07-11-22h-20m-42s` | flat_thin | lid_on_off | IRIS | 317 | y | Pick up the slice of bread from the left side of the counter and place |
| `RAD+62828209+2023-11-26-13h-45m-02s` | flat_thin | reposition | RAD | 1347 | y | Take out two books from the right side of the second shelf on the righ |
| `RAIL+80edfcb1+2023-04-19-15h-12m-55s` | flat_thin | free_form | RAIL | 173 | y | Put the white and orange object onto the orange plate |
| `REAL+4dbb5646+2023-05-17-11h-19m-38s` | flat_thin | concrete | REAL | 370 | y | Open the notebook on the tray |
| `AUTOLab+84bd5053+2023-07-08-09h-31m-28s` | organic | into_container | AUTOLab | 535 | y | Put the orange, yellow and green blocks in the gray pot one at a time |
| `IPRL+2fe9715d+2023-08-24-22h-22m-32s` | organic | fold | IPRL | 394 | y | Place a slice of bread in the toaster and switch it on |
| `IPRL+w026bb9b+2023-09-22-23h-48m-36s` | organic | lid_on_off | IPRL | 187 | y | Put two slices of bread in the toaster |
| `IRIS+226b9f35+2023-07-11-23h-59m-48s` | organic | hang | IRIS | 366 | y | Put the orange cap on the table |
| `IRIS+226b9f35+2023-07-11-23h-56m-26s` | organic | hang | IRIS | 301 | y | Put the red cap on top of the orange cap |
| `PennPAL+acda9df3+2023-06-15-17h-55m-20s` | organic | free_form | PennPAL | 263 | y | Pick up the banana plush toy and put it on top of the toaster |
| `ILIAD+50aee79f+2023-07-12-21h-45m-13s` | rigid_compact | free_form | ILIAD | 676 | y | Put the book on the left in the black box, put the grape toy in the bo |
| `IPRL+2fe9715d+2023-08-21-17h-37m-45s` | rigid_compact | clean_with_cloth | IPRL | 451 | y | Take a tissue out of the box and use it to wipe the table |
| `IPRL+2fe9715d+2023-08-21-17h-34m-50s` | rigid_compact | clean_with_cloth | IPRL | 442 | y | Remove a tissue from the box and put it on the table |
| `RAD+ac111655+2023-09-18-17h-15m-37s` | rigid_compact | reposition | RAD | 2449 | y | Unstack all the toy blocks then put the orange block in the marked zon |
| `RAIL+80edfcb1+2023-06-30-15h-03m-19s` | rigid_compact | free_form | RAIL | 195 | y | Remove the doll from the plate, then put the yellow block on the plate |
| `REAL+4f8ca688+2023-05-31-10h-33m-23s` | rigid_compact | into_container | REAL | 407 | y | Open the lid of the box then put the red cup in the box |
| `REAL+4dbb5646+2023-06-07-12h-03m-19s` | rigid_compact | lid_on_off | REAL | 242 | y | Put the book on the right in the box |
| `GuptaLab+553d1bd5+2023-04-30-12h-57m-10s` | rigid_symmetric | into_container | GuptaLab | 210 | y | Put the marker inside the bowl |
| `IPRL+2fe9715d+2023-08-21-21h-03m-17s` | rigid_symmetric | clean_with_cloth | IPRL | 597 | y | Use the white towel to wipe the table, then take the white lid and put |
| `IRIS+226b9f35+2023-07-11-12h-03m-17s` | rigid_symmetric | free_form | IRIS | 479 | y | Put the black flask on the right side of the countertop and the blue b |
| `IRIS+226b9f35+2023-07-11-12h-01m-54s` | rigid_symmetric | lid_on_off | IRIS | 358 | y | Press the soap bottle nozzle |
| `IRIS+226b9f35+2023-07-11-11h-55m-54s` | rigid_symmetric | open_close | IRIS | 259 | y | Remove the cup from the sink and put it on the countertop near the toa |
| `RAIL+80edfcb1+2023-04-26-16h-06m-56s` | rigid_symmetric | clean_with_cloth | RAIL | 135 | y | Pour the contents of the jug into the bowl  |
| `AUTOLab+44bb9c36+2023-11-28-08h-46m-19s` | rigid_thin | concrete | AUTOLab | 992 | y | Place the green marker on the yellow towel, fold the towel from bottom |
| `AUTOLab+44bb9c36+2023-11-28-09h-00m-36s` | rigid_thin | concrete | AUTOLab | 655 | y | Put the green marker on the towel, fold the towel twice and place it i |
| `GuptaLab+553d1bd5+2023-04-20-12h-38m-54s` | rigid_thin | reposition | GuptaLab | 225 | y | Pick the marker on the counter and put it in the bowl |
| `GuptaLab+553d1bd5+2023-04-20-12h-41m-59s` | rigid_thin | reposition | GuptaLab | 210 | y | Put the marker inside the bowl |
| `IRIS+226b9f35+2023-07-11-21h-54m-44s` | rigid_thin | into_container | IRIS | 543 | y | Place the wooden spoon in the utensil holder |
| `RAD+62828209+2023-09-09-09h-01m-13s` | rigid_thin | reposition | RAD | 602 | y | Pick up the green marker and then put it back down |
| `IPRL+7790ec0a+2023-05-05-17h-22m-57s` | transparent | concrete | IPRL | 194 | y | Pull out the storage bucket slightly under the drawers |
| `IPRL+7790ec0a+2023-05-05-17h-12m-15s` | transparent | concrete | IPRL | 164 | y | Pull the transparent bin out |
| `AUTOLab+44bb9c36+2023-11-28-10h-43m-38s` | unknown_no_annotation | concrete | AUTOLab | 269 | FAIL | Place pen in cloth, wrap pen in cloth, and put in box. |
| `AUTOLab+5d05c5aa+2023-07-26-16h-48m-43s` | unknown_no_annotation | concrete | AUTOLab | 236 | FAIL | insert a cable into a straight custom build channel |
| `CLVR+13759f6e+2023-05-16-15h-05m-43s` | unknown_no_annotation | into_container | CLVR | 253 | FAIL | Move object into or out of container (ex: drawer, clothes hamper, plat |
| `CLVR+13759f6e+2023-05-17-21h-21m-24s` | unknown_no_annotation | free_form | CLVR | 310 | FAIL | Do anything you like that takes multiple steps to complete. |
| `GuptaLab+553d1bd5+2023-04-20-13h-15m-30s` | unknown_no_annotation | into_container | GuptaLab | 176 | FAIL | Move object into or out of container (ex: drawer, clothes hamper, plat |
| `GuptaLab+553d1bd5+2023-05-28-15h-38m-14s` | unknown_no_annotation | open_close | GuptaLab | 307 | FAIL | Open or close slidable objects (ex: toaster, drawers, sliding doors, d |
| `ILIAD+50aee79f+2023-08-18-10h-16m-14s` | unknown_no_annotation | lid_on_off | ILIAD | 530 | FAIL | Move lid on or off of container (ex: pot, cup, pill bottle) |
| `ILIAD+50aee79f+2023-08-30-15h-41m-59s` | unknown_no_annotation | lid_on_off | ILIAD | 152 | FAIL | Move lid on or off of container (ex: pot, cup, pill bottle) |