README.md

---
license: apache-2.0
library_name: lerobot
tags:
  - hil-serl
  - reinforcement-learning
  - sac
  - rlpd
  - robotics
  - ur7e
  - lerobot
pipeline_tag: robotics
datasets:
  - Bigenlight/banana_in_pot_lerobot_v3
---

# HIL-SERL offline prep bundle — Put the right banana in the pot (UR7e)

This repository is the **offline-prepared base for HIL-SERL** (Human-in-the-Loop Sample-Efficient
Robotic reinforcement Learning) on the *"put the right banana in the pot"* task with a
**Universal Robots UR7e**. It contains **everything up to online RL** — the vision reward
classifier, the SAC/RLPD training config, the UR7e kinematics (URDF + FK), the offline demo
buffer builder, and the runbook. The **online phase itself is robot-only** and is not included
here (it requires the physical arm plus `ur_rtde` + `placo` + a gRPC actor/learner).

Built from [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3)
(51 teleop demos / 21,524 frames) using [LeRobot](https://github.com/huggingface/lerobot)'s
HIL-SERL stack.

## Data & hardware setup

| Component | Detail |
|---|---|
| Robot | **Universal Robots UR7e** — 6-DOF collaborative arm, joints in radians. |
| Teleoperation (demos) | **GELLO** low-cost 3D-printed leader arm. |
| Camera 1 | **Intel RealSense D435** — RGB only |
| Camera 2 | **Intel RealSense D435if** — RGB only |
| Camera streams | 1280×720 @ 30 fps, color only (**no depth / IR**). HIL-SERL uses the two RGB views **resized to 128×128**. |
| Task | *"put the right banana in the pot"* — distractors + silver pot; success = correct banana in the pot. |

## What's in this bundle

```
banana_in_pot_hilserl/
├── reward_classifier/
│   └── checkpoint/              # trained success classifier (config.json + model.safetensors)
├── config/
│   └── train_hilserl_ur7e.json  # shared SAC/RLPD learner+actor config (paths relative)
├── build_offline_buffer.py      # rebuilds the 54 MB RL demo buffer from the dataset
├── joint_to_ee.py               # UR7e forward kinematics (Joint → EE), placo-free, validated
├── ur7e.urdf                    # pre-generated UR7e URDF (IK target frame `tool0`)
├── HILSERL_RUNBOOK.md           # exact online start sequence (robot-only steps marked [ROBOT])
└── assets/                      # figures embedded below
```

> **Offline demo buffer is NOT shipped** (it is ~54 MB of video/state transitions). Rebuild it
> from the dataset with `build_offline_buffer.py` — see [Rebuilding the demo buffer](#rebuilding-the-offline-demo-buffer).

## 1. Reward / success classifier

A vision-based binary success detector trained offline, in the exact LeRobot
`reward_classifier` format so it drops straight into HIL-SERL.

- **Encoder:** frozen `lerobot/resnet10` (CNN), per-camera `SpatialLearnedEmbeddings` pool +
  Linear→LayerNorm→Tanh. Following the official LeRobot behavior, the encoder runs under
  `no_grad` so **only the classifier head trains** (~2.36 M trainable params).
- **Cameras:** `observation.images.cam1` + `observation.images.cam2`, 128×128, MEAN_STD-normalized.
- **Labeling** (all 51 demos are successes, so negatives are synthesized): POSITIVE = last 15%
  of frames **and** gripper re-opened (release into pot); NEGATIVE = first 55% of frames; the
  55–85% transport margin is excluded. Split **by episode** (val = eps [4,14,24,34,44]).
- **Deployed with `success_threshold = 0.7`** for release-boundary margin.

| metric | value |
|---|---|
| Val accuracy | **99.49%** |
| Val precision / recall / F1 | 0.979 / 0.991 / 0.985 |
| Train accuracy (balanced) | 100% |
| Confusion (val, thr 0.5) | TP=231, TN=1138, FP=5, FN=2 |

![training curves](assets/training_curves.png)

![confusion matrix](assets/confusion_matrix.png)

## 2. Action spec — EE-delta

The demos are absolute joints, but the HIL-SERL policy acts in **end-effector delta** space.

- **Action = base-frame TCP delta ÷ step_size (0.05 m)**, computed by running **FK on the dataset
  joints** (identical to what online deploy uses as its per-step reference). The gripper is a
  discrete class `{0=close, 1=stay, 2=open}`.
- The offline buffer stores a 4-dim action (continuous xyz + discrete gripper); the SAC critic
  slices `actions[:, :-1]` for the continuous part and a separate discrete critic handles the
  gripper. Hence `output_features["action"].shape = [3]` with `num_discrete_actions = 3`.
- Action stats (21,524 frames): p99 ≈ 0.2 in tanh space, `|Δ|>1` = **0.0%** — comfortably inside
  the `[-1, 1]` range.

### Joint → EE forward kinematics (validated)

`joint_to_ee.py` implements UR7e FK directly from the URDF joint origins (placo-free) and was
validated against the recorded TCP pose:

| subset | pos err median | rot err median |
|---|---|---|
| near-static (‖q̇‖<0.02, n=1884) | **0.85 mm** | **0.16°** |
| all samples (n=42,833) | 28.0 mm | — |

Sub-mm / sub-0.2° error while static confirms the kinematic chain is accurate; the larger
moving-sample error is timing jitter between the two async logging streams (joint vs. TCP),
**not** an FK error. Online deploy IK uses `placo` (Pinocchio), frame `tool0`.

![FK validation](assets/fk_validation.png)

## 3. Training config (SAC + RLPD)

`config/train_hilserl_ur7e.json` is the shared `TrainRLServerPipelineConfig` for the learner and
actor, fully consistent with the offline buffer:

| block | key values |
|---|---|
| `algorithm` (SAC) | `num_critics=2`, `utd_ratio=2`, `discount=0.99`, `temperature_init=0.01`, `grad_clip_norm=10` |
| mixer | `online_offline`, `online_ratio=0.5` (RLPD 50/50 online/offline) |
| `policy` (gaussian_actor) | `vision_encoder=lerobot/resnet10` (frozen), `num_discrete_actions=3`, `online_step_before_learning=100`, `storage_device=cpu` |
| input features | state[7] + cam1[3,128,128] + cam2[3,128,128] |
| output features | `action[3]` (continuous xyz; gripper via discrete head) |
| `env` (gym_manipulator) | `resize_size=[128,128]`, EE `step_sizes=0.05`, IK `urdf=ur7e.urdf`/`tool0`, `reward_classifier` thr 0.7, `robot`/`teleop = null` |

The learner setup was validated end-to-end offline: `python -m lerobot.rl.learner` builds the SAC
policy (2.76 M trainable / 7.67 M total params), loads the offline demo buffer via
`ReplayBuffer.from_lerobot_dataset`, starts its gRPC server, and **idles at the online-buffer gate**
(`len(online_buffer) < online_step_before_learning=100`) waiting for the actor — which is exactly
the "ready for online RL" state.

## 4. The online phase (robot-only, NOT in this repo)

See `HILSERL_RUNBOOK.md` for the exact sequence. In short, on the machine wired to the arm:

1. Install the hardware/transport deps that the offline env intentionally omits: `grpcio`
   (py3.12 wheel), `placo` (IK), `ur_rtde` (UR I/O), `lerobot[hilserl]`.
2. Fill `env.robot` / `env.teleop` in the config (UR7e IP + motors + cameras; gamepad teleop).
3. Tune the workspace placeholders against the real arm: `end_effector_bounds`,
   `fixed_reset_joint_positions`, episode length. **Keep `end_effector_step_sizes = 0.05`** — the
   offline action = TCP-delta ÷ 0.05, so changing it desyncs the demo actions.
4. Start the **learner** (terminal 1) and **actor** (terminal 2) with the same config. The learner
   idles at the gate until the actor supplies ≥100 online transitions, then SAC updates begin with
   the 50/50 RLPD mix.
5. **Human interventions:** gamepad trigger / `space` to take over and give corrective demos; taper
   the intervention rate as the policy improves.

> **RAM note:** the full offline buffer (`from_lerobot_dataset`) materializes every transition
> eagerly and peaks at **~25 GB**. Use ≥48 GB RAM, or subset episodes via
> `--dataset.episodes='[...]'`, or lower `offline_buffer_capacity`.

## Rebuilding the offline demo buffer

The 54 MB RL demo buffer is not shipped. Rebuild it from the LeRobot dataset with the included
script (it converts absolute-joint demos to the EE-delta + reward + done schema):

```bash
python build_offline_buffer.py   # writes ./banana_rl_lerobot (repo_id theo/banana_in_pot_rl)
```

Then point `dataset.root` in `config/train_hilserl_ur7e.json` at the rebuilt directory (the config
ships with a relative `./banana_rl_lerobot`).

## Related repositories

- **Dataset:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3)
- **ACT imitation-learning policy** for the same task:
  [`Bigenlight/act_banana_in_pot`](https://huggingface.co/Bigenlight/act_banana_in_pot)

## Caveats & limitations

- **No true failure episodes.** The classifier's negatives are *early-task* frames (approach/grasp),
  not genuine failed attempts. It reliably separates "task complete" from "task in progress" but has
  never seen a real failure of a completed-looking state — expect over-confidence on OOD near-miss
  end states. **Record a handful of real failure/near-miss episodes early in online HIL-SERL** to
  harden it.
- The excluded 55–85% transport margin means the **decision boundary around the release moment is
  uncalibrated**; `success_threshold=0.7` adds margin.
- **EE safety bounds and reset pose in the config are placeholders** — they MUST be tuned against
  the real arm before letting the policy move.
- Only the classifier head trains (frozen-encoder LeRobot quirk) — fine for this easy visual task,
  the first thing to revisit if a harder task underperforms.
- All validation here is **offline** (config parse → policy/critic build → buffer load → gate). No
  online RL results are included; that is the robot-only next step.