| --- |
| license: apache-2.0 |
| tags: |
| - robotics |
| - lerobot |
| - diffusion-policy |
| - imitation-learning |
| - manipulation |
| --- |
| |
| # Diffusion Policy — banana-in-pot (JOINT, fp16) |
|
|
| Diffusion Policy trained on the **"put the right banana in the pot"** task (UR7e + GELLO |
| teleoperation, 2 RGB cameras), in **JOINT** action space (6 joints + gripper), using |
| **fp16 mixed-precision** training. |
|
|
| - **Checkpoint:** step 80,000 (best open-loop MAE) |
| - **Base library:** [LeRobot](https://github.com/huggingface/lerobot) 0.6.1 (pin `8a74e0a`) |
| - **Dataset:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3) — 51 episodes / 21,524 frames / 30 fps |
| - **fp32 sibling:** [`Bigenlight/diffusion_banana_in_pot_joint`](https://huggingface.co/Bigenlight/diffusion_banana_in_pot_joint) |
|
|
| ## Architecture |
|
|
| ResNet18 (ImageNet-pretrained, trained end-to-end) + SpatialSoftmax vision encoder → |
| FiLM-conditioned 1D temporal U-Net denoiser (DDPM-100, epsilon prediction). ~278M params. |
| Images resized to 360×640, crop off. `n_obs_steps=2`, `horizon=64`, `n_action_steps=32`. |
|
|
| ## Training |
|
|
| - **Precision:** fp16 via HF Accelerate `mixed_precision=fp16` (automatic GradScaler). |
| - Requires a `dtype` field on `DiffusionConfig` (absent upstream at this pin); launched with |
| `--policy.dtype=float16`. |
| - Batch 8, 80k steps, AdamW, seed 1000, 45 train / 6 held-out episodes. |
| - **Hardware:** single RTX A4000. **~4.48 step/s, ~4.96 h** (vs ~3.49 step/s / ~6.4 h fp32 → |
| ~1.25× faster, ~22% less wall-clock). No NaN/instability. |
|
|
| ## Open-loop evaluation (DDIM-10, held-out episodes 45–50) |
|
|
| | | fp32 @80k | **fp16 @80k** | |
| |---|---|---| |
| | poseMAE (rad) | 0.0845 | **0.08268** | |
| | gripper accuracy | 0.953 | **0.960** | |
|
|
| **fp16 matches (marginally beats, within noise) fp32 accuracy at ~22% lower training cost.** |
| Select the deploy checkpoint by open-loop MAE, **not** `eval_loss` (which rises during |
| training for diffusion policies without indicating overfitting). |
|
|
| ## Intended use & limitations |
|
|
| Research artifact. Small single-task, single-scene, real-world (noisy) dataset of 51 |
| success-only demonstrations; offline metrics only — no closed-loop hardware success rate |
| measured yet. Not safety-validated for autonomous operation. |
|
|
