---
license: apache-2.0
library_name: pytorch
pipeline_tag: robotics
tags:
  - robotics
  - manipulation
  - dual-arm
  - action-chunking
  - act
  - behavior-cloning
  - mujoco
datasets:
  - Physical41/lift_classical_sim_v1
---

# lift_act_sim_v2 — the no-foam cell

An action chunking transformer trained by behaviour cloning to lift a 2 × 6 × 2 cm red block 10 cm
off the table of a **simulated** Dobot X-Trainer dual-arm cell — `omn4i/Lift-v0` in
[`Physical41/omn4i_envs`](https://github.com/Physical41/omn4i_envs).

**81.0% success on held-out seeds, against the scripted expert's 99.0%.**

This is published as a **baseline to beat, not a good model.** It has a 23-point gap between the
arms and its cause is not yet known. See "The open problem" below.

## Which cell this is for — read this before using either version

| | `lift_act_sim_v1` | **`lift_act_sim_v2` (this one)** |
|---|---|---|
| Cell | 50 mm **foam slab** on the table | **no foam** — the cell as it is now |
| Trained at | pre-foam-removal tree | commit `ebb10c8` (`main` HEAD) |
| Corpus | 2847 episodes | 1987 episodes |
| Overall | 93.8% | **81.0%** |
| Left / right arm | 93.8% / 93.8% | **93.0% / 69.8%** |

**`v1` is for a cell that no longer exists.** The foam was removed from `main` in `326dc18`, which
changed every object height by 50 mm and gave the arms back 50 mm of reach. `v1`'s 93.8% is a real
measurement, but of a configuration you cannot reproduce from `main` today. **Use v2 with current
`main`.**

The weights here were verified to load and run under `main`'s own `omn4i_envs/baselines/act.py` at
`ebb10c8`. Every file defining the lift task — `lift.py`, `env.py`, `scene.xml`, `lift.xml`, both
arm XMLs, `contract.py`, `classical.py`, `benchmark.py`, `transforms.py` — is byte-identical
between `main` and the tree this was trained on.

## Results

500 held-out episodes (`benchmark.TEST_SEEDS`, `range(10000, 10500)`), by
`python tools/benchmark.py act`. None of these seeds was trained on.

| | `act` (this model) | `classical` (its teacher) |
|---|---|---|
| **Success** | **81.0%** (405/500) | 99.0% (495/500) |
| 95% Wilson interval | 77.3% – 84.2% | 97.7% – 99.6% |
| Per-step latency, median → p99 | 7.52 → 7.61 ms | 0.01 → 0.01 ms |

Against a 33 ms environment step the policy uses 7.5 ms. It replans every step.

### By arm and band — where it goes wrong

| Band | left | right |
|---|---|---|
| under 0.40 m | 95% (52/55) | 95% (42/44) |
| 0.40 to 0.50 m | 92% (78/85) | 74% (62/84) |
| **0.50 to 0.62 m** | 94% (80/85) | **54% (62/115)** |
| 0.62 m and out | 88% (15/17) | 93% (14/15) |
| **overall** | **93.0%** | **69.8%** |

The expert is symmetric on the same seeds (98.8% left, 99.2% right), so nothing about the task
favours one side.

## The open problem

**The right arm is 23 points behind the left and we do not know why.**

What it is **not**:

- **Not data scarcity.** The corpus is balanced by arm — 978 left-arm episodes against 1009 right.
  The worst cell, right 0.50–0.62 m at 54%, has *more* training episodes (461) than its left-hand
  counterpart (407) which scores 94%.
- **Not the two demonstration defects fixed in `main` PR #13** (the idle arm swinging 90° every
  episode, and the wrist roll scattering across its equivalents). Both are verified absent from
  this corpus — idle drift 0.3°, wrist roll within ±90° on both arms. Fixing them *coincided* with
  the gap closing on the foam cell, but the gap returned when only the cell changed, so they were
  not the whole cause.
- **Not the expert.** `classical` scores 99.2% on right-arm seeds.

The open lead is the **right arm's joint 6 sign, which is unverified upstream**. It turns about the
tool axis, which the end-effector site sits on, so no trajectory data can settle it — but it does
set which way the jaws straddle the block. That is consistent with a policy that reaches correctly
and closes on nothing, and with the expert being unaffected, since its inverse kinematics is
self-consistent inside the same model.

## Architecture and training

18.7M parameters. One **shared** ResNet18 across the three cameras, feeding a 4-layer
encoder/decoder at width 256, 8 heads. Predicts 32 actions from one observation, L1 loss, temporal
ensembling at inference. The backbone starts from ImageNet weights and is **fine-tuned** at 1e-5
against 1e-4 for the rest — 11.2M of the 18.7M parameters, not frozen.

No CVAE (the demonstrations come from a deterministic planner) and no language conditioning (one
task, so the prompt is a constant).

1900 demonstrations / 117,874 frames from `Physical41/lift_classical_sim_v1`, 87 episodes held out
stratified by reach band. 40,000 steps at batch 64, 59.4 minutes on one RTX 4090. Final held-out
chunk error 0.01123 rad.

Note that held-out chunk error has been a **poor predictor of rollout success** on this task: a run
at 0.01138 scored 93.4% and one at 0.00928 scored 73.2%. Judge by the benchmark, not by the loss.

## Loading

```python
from omn4i_envs.baselines import act          # import before mujoco
policy = act.ACTPolicy(checkpoint="act.pt")

observation, info = env.reset(seed=10000)
policy.reset(observation, info)
action = policy(observation)                  # (14,) float32, clamped into the action space
```

Actions are absolute joint position targets in **the controller's convention, which differs per
arm** — the right is read at the negation of the left, and neither neutral is the zero vector.
Gripper entries are normalized **1.0 open, 0.0 closed**.

## This is simulation only

Recorded real joint angles do not reproduce the real robot's kinematics in this simulation, and the
joint sign and zero-offset convention between the vendor stack and the MJCF is only partly
resolved — the right arm's joint 6 sign remains open, which may well be the same thing causing the
asymmetry above. **Do not cite these numbers as evidence about the physical robot.**