---
license: apache-2.0
library_name: pytorch
pipeline_tag: robotics
tags:
  - robotics
  - manipulation
  - dual-arm
  - action-chunking
  - act
  - behavior-cloning
  - mujoco
datasets:
  - Physical41/lift_classical_sim_v1
---

# lift_act_sim_v3 — the cell as the lab measured it

An action chunking transformer trained by behaviour cloning to lift a 2 × 6 × 2 cm red block
10 cm off the table of a **simulated** Dobot X-Trainer dual-arm cell — `omn4i/Lift-v0` in
[`Physical41/stack`](https://github.com/Physical41/stack).

**88.6% success on held-out seeds, against the scripted expert's 99.0%.**

## Which cell this is for — read this before using any version

| | `lift_act_sim_v1` | `lift_act_sim_v2` | **`v3` (this one)** |
|---|---|---|---|
| Cell | 50 mm foam slab on the table | foam removed | **geometry realigned to the lab's measured calibration** |
| Trained at | pre-foam-removal tree | `ebb10c8` (`omn4i_envs`) | **`648ecd5` (`Physical41/stack`)** |
| Corpus | 2847 episodes | 1987 episodes | **1985 episodes** |
| Overall | 93.8% | 81.0% | **88.6%** |
| Left / right arm | 93.8% / 93.8% | 93.0% / 69.8% | **96.3% / 81.2%** |

**v1 and v2 are both for cells that no longer exist.** Use v3 with current `main`.

### What changed under v2, and what it cost

The simulation was reading its own numbers where the lab had measured them. Five corrections
landed in `Physical41/stack@7de2fcc`; three of them move pixels:

| | simulation had | the lab measured |
|---|---|---|
| table surface | a round −10 mm | **+3.11 mm**, from 48 fingertip contacts |
| arm separation | 1060.0 mm | **1065.4 mm**, load-cell fit at 3.1 mm RMS |
| right arm riser | 1069.4 mm | — it stood 9.4 mm off its own post |
| `ee_pose` frame | MuJoCo world | the controller's frame — a **half turn and 223.4 mm** apart |

**v2 scores 50.0% on this cell**, against the 81.0% its own card reports on the cell it was
trained for. Measured here on the same 500 seeds, same tree, same torch:

| | overall | left | right |
|---|---|---|---|
| v2 on the realigned cell | 50.0% (250/500) | 53.7% | 46.5% |
| **v3 on the realigned cell** | **88.6% (443/500)** | **96.3%** | **81.2%** |

Attribution, by reverting one change at a time: the table and riser cost ~22 points, the 5.4 mm
of arm movement ~8.7, and the `ee_pose` frame change **exactly zero** — which is a clean
confirmation that this architecture reads only images and joint positions. The regression is
entirely visual. Five millimetres of arm movement costing nearly nine points is the number worth
remembering about behaviour cloning.

## Results

500 held-out episodes (`benchmark.TEST_SEEDS`, `range(10000, 10500)`). None was trained on:
the corpus is seeds 0–1999, from `benchmark.DEV_SEEDS`.

| | success | 95% Wilson | left arm | right arm |
|---|---|---|---|---|
| **`act` (this model)** | **88.6%** (443/500) | 85.7% – 91.3% | 96.3% (235/244) | 81.2% (208/256) |
| `classical` (its teacher) | 99.0% (495/500) | 97.7% – 99.6% | — | — |

### Do not read 88.6% against v2's 81.0% as an improvement

They are different cells. v2's 81.0% was measured on the pre-realignment geometry, and no
ablation has been run to attribute the difference. Everything controllable was held equal —
1985 episodes against 1987, 40,000 steps, the same hyperparameters and the same expert — but
the environment itself moved, so the honest claim is *the regression is repaired*, not *the
model is better*. The only controlled comparison on this page is 50.0% against 88.6%, on one
cell.

## The open problem, still open

**A 15.1 point gap between the arms**, and its cause is still not known. It narrowed from
v2's 23.2 points, which is also unexplained — the same caveat applies, since the cell changed
underneath it. The expert shows no such gap: it solves both arms at 99%. Whatever this is, it
is the model's, not the demonstrations'.

## Training

```
python sim/tools/record.py lift <corpus> --episodes 2000
python sim/tools/train_act.py <corpus> --steps 40000
```

| | |
|---|---|
| Corpus | 1985 of 2000 dev seeds succeeded, 104,678 frames, 5955 videos at 30 fps |
| Parameters | 18.7 M |
| Steps / batch / chunk | 40,000 / 48 / 32 |
| Held-out mean absolute error | 0.00714 rad |
| Wall clock | 30 min recording, 37.5 min training, one RTX PRO 6000 |

Recorded and trained in a clean checkout of `648ecd5`, so the commit in the table above
describes exactly what produced these weights.

## Inputs

Three cameras (`overhead`, `left_wrist`, `right_wrist`) at 240 × 320, and the 14 measured joint
positions. **It does not read `ee_pose` or depth.** That is why the frame correction cost it
nothing, and it is worth knowing before assuming a pose fix will reach it.

## Using it

```python
from omn4i_envs.baselines import act
policy = act.load("act.pt", replan=1)()

observation, info = env.reset(seed=10000)
policy.reset(observation, info)
action = policy(observation)
```

Or watch it:

```bash
python sim/examples/watch_task.py --task lift --checkpoint act.pt
```

which defaults to the test seeds, because a policy shown its own training seeds is
demonstrating memory.