| --- |
| library_name: lerobot |
| license: apache-2.0 |
| pipeline_tag: robotics |
| tags: |
| - act |
| - diffusion |
| - robotics |
| - imitation-learning |
| - behavior-cloning |
| - aloha |
| - pytorch_model_hub_mixin |
| - model_hub_mixin |
| datasets: |
| - JHeisler/aloha_solo_left_4_6_26 |
| --- |
| |
| # Hybrid ACT+Diffusion — ALOHA Single-Arm (Left) — 13.4k steps |
|
|
| Custom **HybridACTDiffusion** policy: ACT visual encoder (ResNet18 + 4-layer Transformer, mean-pooled) feeds a Diffusion U-Net decoder (FiLM conditioning, DDPM training, DDIM 10-step inference). No VAE — diffusion handles multimodal action distributions directly. |
|
|
| This is the **initial 13.4k-step Hybrid baseline (S002)**. For the longer 40k retrain, see [JHeisler/aloha_solo_left_act_diffusion_40k](https://huggingface.co/JHeisler/aloha_solo_left_act_diffusion_40k). |
|
|
| ## Architecture |
|
|
| ``` |
| Images (cam_high, cam_left_wrist) + State (dim=9) |
| │ |
| ▼ |
| ACT Encoder (ResNet18 → 4-layer Transformer) → mean-pool → (B, 512) global cond vector |
| │ |
| ▼ |
| Diffusion U-Net (DiffusionConditionalUnet1d, FiLM modulation, down_dims=(256,512)) |
| │ DDPM training / DDIM 10-step inference |
| ▼ |
| Action chunks (chunk_size=100, action_dim=9) |
| ``` |
|
|
| ## Training Config |
|
|
| | Field | Value | |
| |---|---| |
| | Architecture | HybridACTDiffusion (ACT encoder + Diffusion U-Net) — see `lerobot/common/policies/hybrid_act_diffusion/` | |
| | Dataset | [JHeisler/aloha_solo_left_4_6_26](https://huggingface.co/datasets/JHeisler/aloha_solo_left_4_6_26) — 50 episodes, 29,785 samples, 30 fps | |
| | State / action dim | 9 / 9 | |
| | Cameras | `cam_high`, `cam_left_wrist` (3×480×640 each) | |
| | Steps | 13,400 | |
| | Batch size | 24 (DOE winner) | |
| | Learning rate | 3e-5 | |
| | Total samples seen | ~321K (~10.6 epochs) | |
| | AMP | enabled | |
| | torch.compile | enabled | |
| | Diffusion scheduler | DDPM training (100 timesteps, squaredcos_cap_v2), DDIM at inference (10 steps) | |
| | Final loss (DDPM noise-pred MSE) | 0.011–0.020 | |
| | Final grad norm | 0.2–0.7 | |
| | Wall clock | ~1h 16min on RTX A4500 | |
| | LeRobot pin | `96c7052777aca85d4e55dfba8f81586103ba8f61` (with custom hybrid_act_diffusion policy added) | |
|
|
| ## Project Lineage |
|
|
| | Workstream | Model | Steps | Samples | HF | |
| |---|---|---|---|---| |
| | S001 | ACT | 13,400 | 640K | [act_left](https://huggingface.co/JHeisler/aloha_solo_left_4_6_26_act_left) | |
| | **S002** | **Hybrid ACT+Diffusion** | **13,400** | **321K** | **this repo** | |
| | S003 | ACT (shipped) | 40,000 | 1.92M | [act_left_40k](https://huggingface.co/JHeisler/aloha_solo_left_4_6_26_act_left_40k) | |
| | S004 | Hybrid ACT+Diffusion | 40,000 | 1.12M | [act_diffusion_40k](https://huggingface.co/JHeisler/aloha_solo_left_act_diffusion_40k) | |
|
|
| ## Notes on loss comparability |
|
|
| DDPM noise-prediction MSE (this model) and ACT's L1+KL combo (S001/S003) are different loss surfaces — absolute loss values are NOT directly comparable across architectures. The right comparison is offline action L1 on held-out episodes or real-robot rollout success rate. |
|
|
| ## Usage |
|
|
| The custom policy class lives in this project's LeRobot fork. To use: |
|
|
| ```python |
| # Requires lerobot pinned to 96c7052 with hybrid_act_diffusion policy package added |
| from lerobot.common.policies.hybrid_act_diffusion.modeling_hybrid_act_diffusion import HybridACTDiffusionPolicy |
| policy = HybridACTDiffusionPolicy.from_pretrained("JHeisler/aloha_solo_left_act_diffusion") |
| ``` |
|
|
| ## Citation / Course |
|
|
| EN.525.681 school project — JHU Whiting School of Engineering. Team: Jake Heisler, Laura Kroening, Purushottam Shukla. |
|
|
| Code reference: [HuggingFace LeRobot](https://github.com/huggingface/lerobot) at commit `96c7052` with custom hybrid policy package. |
|
|
