| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - manipulation |
| - dual-arm |
| - action-chunking |
| - act |
| - behavior-cloning |
| - mujoco |
| datasets: |
| - Physical41/lift_classical_sim_v1 |
| --- |
| |
| # lift_act_sim_v3 — the cell as the lab measured it |
| |
| An action chunking transformer trained by behaviour cloning to lift a 2 × 6 × 2 cm red block |
| 10 cm off the table of a **simulated** Dobot X-Trainer dual-arm cell — `omn4i/Lift-v0` in |
| [`Physical41/stack`](https://github.com/Physical41/stack). |
| |
| **88.6% success on held-out seeds, against the scripted expert's 99.0%.** |
| |
| ## Which cell this is for — read this before using any version |
| |
| | | `lift_act_sim_v1` | `lift_act_sim_v2` | **`v3` (this one)** | |
| |---|---|---|---| |
| | Cell | 50 mm foam slab on the table | foam removed | **geometry realigned to the lab's measured calibration** | |
| | Trained at | pre-foam-removal tree | `ebb10c8` (`omn4i_envs`) | **`648ecd5` (`Physical41/stack`)** | |
| | Corpus | 2847 episodes | 1987 episodes | **1985 episodes** | |
| | Overall | 93.8% | 81.0% | **88.6%** | |
| | Left / right arm | 93.8% / 93.8% | 93.0% / 69.8% | **96.3% / 81.2%** | |
|
|
| **v1 and v2 are both for cells that no longer exist.** Use v3 with current `main`. |
|
|
| ### What changed under v2, and what it cost |
|
|
| The simulation was reading its own numbers where the lab had measured them. Five corrections |
| landed in `Physical41/stack@7de2fcc`; three of them move pixels: |
|
|
| | | simulation had | the lab measured | |
| |---|---|---| |
| | table surface | a round −10 mm | **+3.11 mm**, from 48 fingertip contacts | |
| | arm separation | 1060.0 mm | **1065.4 mm**, load-cell fit at 3.1 mm RMS | |
| | right arm riser | 1069.4 mm | — it stood 9.4 mm off its own post | |
| | `ee_pose` frame | MuJoCo world | the controller's frame — a **half turn and 223.4 mm** apart | |
|
|
| **v2 scores 50.0% on this cell**, against the 81.0% its own card reports on the cell it was |
| trained for. Measured here on the same 500 seeds, same tree, same torch: |
|
|
| | | overall | left | right | |
| |---|---|---|---| |
| | v2 on the realigned cell | 50.0% (250/500) | 53.7% | 46.5% | |
| | **v3 on the realigned cell** | **88.6% (443/500)** | **96.3%** | **81.2%** | |
|
|
| Attribution, by reverting one change at a time: the table and riser cost ~22 points, the 5.4 mm |
| of arm movement ~8.7, and the `ee_pose` frame change **exactly zero** — which is a clean |
| confirmation that this architecture reads only images and joint positions. The regression is |
| entirely visual. Five millimetres of arm movement costing nearly nine points is the number worth |
| remembering about behaviour cloning. |
|
|
| ## Results |
|
|
| 500 held-out episodes (`benchmark.TEST_SEEDS`, `range(10000, 10500)`). None was trained on: |
| the corpus is seeds 0–1999, from `benchmark.DEV_SEEDS`. |
|
|
| | | success | 95% Wilson | left arm | right arm | |
| |---|---|---|---|---| |
| | **`act` (this model)** | **88.6%** (443/500) | 85.7% – 91.3% | 96.3% (235/244) | 81.2% (208/256) | |
| | `classical` (its teacher) | 99.0% (495/500) | 97.7% – 99.6% | — | — | |
|
|
| ### Do not read 88.6% against v2's 81.0% as an improvement |
|
|
| They are different cells. v2's 81.0% was measured on the pre-realignment geometry, and no |
| ablation has been run to attribute the difference. Everything controllable was held equal — |
| 1985 episodes against 1987, 40,000 steps, the same hyperparameters and the same expert — but |
| the environment itself moved, so the honest claim is *the regression is repaired*, not *the |
| model is better*. The only controlled comparison on this page is 50.0% against 88.6%, on one |
| cell. |
|
|
| ## The open problem, still open |
|
|
| **A 15.1 point gap between the arms**, and its cause is still not known. It narrowed from |
| v2's 23.2 points, which is also unexplained — the same caveat applies, since the cell changed |
| underneath it. The expert shows no such gap: it solves both arms at 99%. Whatever this is, it |
| is the model's, not the demonstrations'. |
|
|
| ## Training |
|
|
| ``` |
| python sim/tools/record.py lift <corpus> --episodes 2000 |
| python sim/tools/train_act.py <corpus> --steps 40000 |
| ``` |
|
|
| | | | |
| |---|---| |
| | Corpus | 1985 of 2000 dev seeds succeeded, 104,678 frames, 5955 videos at 30 fps | |
| | Parameters | 18.7 M | |
| | Steps / batch / chunk | 40,000 / 48 / 32 | |
| | Held-out mean absolute error | 0.00714 rad | |
| | Wall clock | 30 min recording, 37.5 min training, one RTX PRO 6000 | |
|
|
| Recorded and trained in a clean checkout of `648ecd5`, so the commit in the table above |
| describes exactly what produced these weights. |
|
|
| ## Inputs |
|
|
| Three cameras (`overhead`, `left_wrist`, `right_wrist`) at 240 × 320, and the 14 measured joint |
| positions. **It does not read `ee_pose` or depth.** That is why the frame correction cost it |
| nothing, and it is worth knowing before assuming a pose fix will reach it. |
| |
| ## Using it |
| |
| ```python |
| from omn4i_envs.baselines import act |
| policy = act.load("act.pt", replan=1)() |
| |
| observation, info = env.reset(seed=10000) |
| policy.reset(observation, info) |
| action = policy(observation) |
| ``` |
| |
| Or watch it: |
| |
| ```bash |
| python sim/examples/watch_task.py --task lift --checkpoint act.pt |
| ``` |
| |
| which defaults to the test seeds, because a policy shown its own training seeds is |
| demonstrating memory. |
| |
