Add files using upload-large-folder tool

README.md

@@ -0,0 +1,130 @@
+---
+license: apache-2.0
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+  - robotics
+  - diffusion-policy
+  - lerobot
+  - so-101
+  - so101
+  - pick-orange
+  - leisaac
+datasets:
+  - LightwheelAI/leisaac-pick-orange
+---
+
+# DiffusionPolicy — LeIsaac SO-101 PickOrange
+
+From-scratch [Diffusion Policy](https://arxiv.org/abs/2303.04137) fine-tune on
+[`LightwheelAI/leisaac-pick-orange`](https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange)
+for the LeIsaac `LeIsaac-SO101-PickOrange-v0` task in Isaac Sim (SO-101 leader arm,
+front + wrist cameras, three oranges → plate).
+
+## Headline result
+
+| Eval budget | Episodes | Success rate |
+| --- | --- | --- |
+| 3 × 60 s | 3 | **1/3** (Episode 2: three oranges placed + arm returned to rest pose) |
+| 3 × 180 s | 3 | _pending — see [Results](#results)_ |
+
+Compared to other policies trained on the same 60 ep × 36 k frame dataset:
+
+| Policy | Params | Eval window | Result |
+| --- | --- | --- | --- |
+| ACT ([`shadowHokage/act_policy`](https://huggingface.co/shadowHokage/act_policy)) | ~80 M | 60 s | ✅ 1/1 |
+| GR00T N1.5 ([`LightwheelAI/leisaac-pick-orange-v0`](https://huggingface.co/LightwheelAI/leisaac-pick-orange-v0)) | ~3 B | 60 s | ✅ 1/1 |
+| SmolVLA v1 (this repo's local fine-tune) | ~450 M | 120 s | 🟡 2/5 |
+| **Diffusion Policy (this ckpt @ 100 k)** | **267 M** | **60 s** | **🟢 1/3** |
+
+## Training configuration
+
+- **Base**: from scratch — no pre-trained backbone
+- **Vision backbone**: ResNet18, `use_group_norm=True`, group norm replaces batch norm
+- **Diffusion**: DDPM, 100 timesteps train + inference, ε-prediction
+- **Architecture**: UNet down_dims `[512, 1024, 2048]`, `n_obs_steps=2`, `horizon=16`, `n_action_steps=8`
+- **Image preproc**: input resolution 480×640, internally resized to 240×320 (`policy.resize_shape=[240, 320]`)
+- **Cameras**: `observation.images.front`, `observation.images.wrist`
+- **State**: 6 joints (5 arm + 1 gripper)
+- **Optimizer**: Adam, lr 1e-4, weight_decay 1e-6, betas (0.95, 0.999)
+- **Scheduler**: cosine with 500 warmup steps
+- **Batch**: 32 per device
+- **Steps**: 100,000 (resume @ 60k saved ckpt → continued to 100k under same config)
+- **Time on 1× RTX 3090 24 GB**: ~8 h total, ~2.5 step/s steady-state
+- **Video backend**: pyav (torchcodec + 4 workers segfaults on long runs)
+- **Wandb**: disabled
+
+Training driver: [`scripts/finetune/lerobot_finetune.sh`](https://github.com/vitorcen/LeIsaac/blob/main/scripts/finetune/lerobot_finetune.sh)
+in [`vitorcen/LeIsaac`](https://github.com/vitorcen/LeIsaac), invoked as
+
+```bash
+POLICY_TYPE=diffusion \
+DATASET_REPO_ID=LightwheelAI/leisaac-pick-orange \
+OUTPUT_NAME=dp-leisaac-pick-orange-v2 \
+STEPS=100000 BATCH_SIZE=32 NUM_WORKERS=2 SAVE_FREQ=20000 \
+EXTRA_ARGS='--dataset.video_backend=pyav --policy.resize_shape=[240,320]' \
+bash scripts/finetune/lerobot_finetune.sh
+```
+
+## Inference
+
+This ckpt is served via the LeRobot async-inference policy server. Two upstream
+patches are required for Diffusion Policy to work through that path (the server
+calls `predict_action_chunk()` instead of `select_action()`):
+
+1. `predict_action_chunk()` made self-populating — pops the rollout-side `ACTION`
+   stub from the input batch, stacks raw image keys into `OBS_IMAGES`, and feeds
+   the current observation into the policy's observation queue (auto-replicates
+   to fill `n_obs_steps` on first call). Without this the server hits either
+   `torch.stack([])` (empty deque) or `NoneType` (rollout `action=None` populated
+   into the action deque).
+2. Server-side traceback logging in `Error in StreamActions` for debuggability.
+
+Both patches live in [`vitorcen/lerobot`](https://github.com/vitorcen/lerobot)
+on top of LeRobot ≥ 0.5. Without them the server logs
+`Error in StreamActions: stack expects a non-empty TensorList` and never
+produces an action.
+
+End-to-end eval recipe (Isaac Sim + LeIsaac client):
+
+```bash
+# 1) Start LeRobot async-inference server
+bash scripts/policy_server.sh start lerobot
+
+# 2) Run policy eval against this ckpt
+cd LeIsaac && \
+  PYTHONUNBUFFERED=1 python -u scripts/evaluation/policy_inference.py \
+    --task=LeIsaac-SO101-PickOrange-v0 \
+    --eval_rounds=3 --episode_length_s=60 \
+    --policy_type=lerobot-diffusion \
+    --policy_host=127.0.0.1 --policy_port=8080 \
+    --policy_timeout_ms=30000 \
+    --policy_language_instruction='Pick the orange to the plate' \
+    --policy_checkpoint_path=<path-to>/pretrained_model \
+    --policy_action_horizon=8 --device=cuda --enable_cameras
+```
+
+## Results
+
+Success criterion (strict, defined in
+[`pick_orange/mdp/terminations.py:task_done`](https://github.com/vitorcen/LeIsaac/blob/main/source/leisaac/leisaac/tasks/pick_orange/mdp/terminations.py)):
+**all three oranges within plate x/y/z range AND the arm returned to rest
+pose**.
+
+- @ 60 k step: 0 / 3 success at 60 s, but Round 3 placed 2 of 3 oranges before
+  the timeout — capability was real, time/robustness budget was not.
+- @ 100 k step: **1 / 3 success at 60 s** (Round 2: three oranges placed + rest
+  pose within 60 s). The other two rounds time out before all conditions are
+  met.
+
+The success window is genuinely tight at 60 s for a 3-object pick-and-place
+plus rest-pose recovery; a longer episode budget will likely improve the
+reported rate.
+
+## License & attribution
+
+- Code & weights: Apache-2.0
+- Dataset: [`LightwheelAI/leisaac-pick-orange`](https://huggingface.co/datasets/LightwheelAI/leisaac-pick-orange)
+  (Apache-2.0)
+- Policy class: LeRobot `--policy.type=diffusion`, Columbia Artificial
+  Intelligence, Robotics Lab et al.

config.json

@@ -0,0 +1,95 @@
+{
+    "type": "diffusion",
+    "n_obs_steps": 2,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.wrist": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "use_peft": false,
+    "push_to_hub": false,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": "/home/david/work/LeIsaac/outputs/dp-leisaac-pick-orange-v2/checkpoints/060000/pretrained_model",
+    "horizon": 16,
+    "n_action_steps": 8,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MIN_MAX",
+        "ACTION": "MIN_MAX"
+    },
+    "drop_n_last_frames": 7,
+    "vision_backbone": "resnet18",
+    "resize_shape": [
+        240,
+        320
+    ],
+    "crop_ratio": 1.0,
+    "crop_shape": null,
+    "crop_is_random": true,
+    "pretrained_backbone_weights": null,
+    "use_group_norm": true,
+    "spatial_softmax_num_keypoints": 32,
+    "use_separate_rgb_encoder_per_camera": false,
+    "down_dims": [
+        512,
+        1024,
+        2048
+    ],
+    "kernel_size": 5,
+    "n_groups": 8,
+    "diffusion_step_embed_dim": 128,
+    "use_film_scale_modulation": true,
+    "noise_scheduler_type": "DDPM",
+    "num_train_timesteps": 100,
+    "beta_schedule": "squaredcos_cap_v2",
+    "beta_start": 0.0001,
+    "beta_end": 0.02,
+    "prediction_type": "epsilon",
+    "clip_sample": true,
+    "clip_sample_range": 1.0,
+    "num_inference_steps": null,
+    "compile_model": false,
+    "compile_mode": "reduce-overhead",
+    "do_mask_loss_for_padding": false,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.95,
+        0.999
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-06,
+    "scheduler_name": "cosine",
+    "scheduler_warmup_steps": 500
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f20edc59961b09e28a9233653a2cd16a81e5417e5c90f21a9afa03eeb4bf35d
+size 1066516952

policy_postprocessor.json

@@ -0,0 +1,32 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b9214914398156f17ddf08a5082c20b24efb70a7ba7677e8bc3d3ff68a722db
+size 3756

policy_preprocessor.json

@@ -0,0 +1,64 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.wrist": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_3_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b9214914398156f17ddf08a5082c20b24efb70a7ba7677e8bc3d3ff68a722db
+size 3756

train_config.json

@@ -0,0 +1,234 @@
+{
+    "dataset": {
+        "repo_id": "LightwheelAI/leisaac-pick-orange",
+        "root": "/home/david/work/LeIsaac/datasets/raw/leisaac-pick-orange",
+        "episodes": null,
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": null,
+        "use_imagenet_stats": true,
+        "video_backend": "pyav",
+        "return_uint8": false,
+        "streaming": false
+    },
+    "env": null,
+    "policy": {
+        "type": "diffusion",
+        "n_obs_steps": 2,
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.images.front": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            },
+            "observation.images.wrist": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    6
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": false,
+        "use_peft": false,
+        "push_to_hub": false,
+        "repo_id": null,
+        "private": null,
+        "tags": null,
+        "license": null,
+        "pretrained_path": "/home/david/work/LeIsaac/outputs/dp-leisaac-pick-orange-v2/checkpoints/060000/pretrained_model",
+        "horizon": 16,
+        "n_action_steps": 8,
+        "normalization_mapping": {
+            "VISUAL": "MEAN_STD",
+            "STATE": "MIN_MAX",
+            "ACTION": "MIN_MAX"
+        },
+        "drop_n_last_frames": 7,
+        "vision_backbone": "resnet18",
+        "resize_shape": [
+            240,
+            320
+        ],
+        "crop_ratio": 1.0,
+        "crop_shape": null,
+        "crop_is_random": true,
+        "pretrained_backbone_weights": null,
+        "use_group_norm": true,
+        "spatial_softmax_num_keypoints": 32,
+        "use_separate_rgb_encoder_per_camera": false,
+        "down_dims": [
+            512,
+            1024,
+            2048
+        ],
+        "kernel_size": 5,
+        "n_groups": 8,
+        "diffusion_step_embed_dim": 128,
+        "use_film_scale_modulation": true,
+        "noise_scheduler_type": "DDPM",
+        "num_train_timesteps": 100,
+        "beta_schedule": "squaredcos_cap_v2",
+        "beta_start": 0.0001,
+        "beta_end": 0.02,
+        "prediction_type": "epsilon",
+        "clip_sample": true,
+        "clip_sample_range": 1.0,
+        "num_inference_steps": null,
+        "compile_model": false,
+        "compile_mode": "reduce-overhead",
+        "do_mask_loss_for_padding": false,
+        "optimizer_lr": 0.0001,
+        "optimizer_betas": [
+            0.95,
+            0.999
+        ],
+        "optimizer_eps": 1e-08,
+        "optimizer_weight_decay": 1e-06,
+        "scheduler_name": "cosine",
+        "scheduler_warmup_steps": 500
+    },
+    "output_dir": "/home/david/work/LeIsaac/outputs/dp-leisaac-pick-orange-v2",
+    "job_name": "diffusion",
+    "resume": true,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 2,
+    "batch_size": 32,
+    "prefetch_factor": 4,
+    "persistent_workers": true,
+    "steps": 100000,
+    "eval_freq": 20000,
+    "log_freq": 200,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 20000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adam",
+        "lr": 0.0001,
+        "weight_decay": 1e-06,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.95,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": {
+        "type": "diffuser",
+        "num_warmup_steps": 500,
+        "name": "cosine"
+    },
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 22,
+        "use_async_envs": true
+    },
+    "wandb": {
+        "enable": false,
+        "disable_artifact": false,
+        "project": "lerobot",
+        "entity": null,
+        "notes": null,
+        "run_id": null,
+        "mode": null,
+        "add_tags": true
+    },
+    "peft": null,
+    "use_rabc": false,
+    "rabc_progress_path": null,
+    "rabc_kappa": 0.01,
+    "rabc_epsilon": 1e-06,
+    "rabc_head_mode": "sparse",
+    "rename_map": {},
+    "checkpoint_path": "/home/david/work/LeIsaac/outputs/dp-leisaac-pick-orange-v2/checkpoints/060000"
+}