Upload sensor diffusion model - 60 epochs completed

README.md

@@ -0,0 +1,146 @@
+---
+license: mit
+tags:
+- robotics
+- imitation-learning
+- diffusion-policy
+- manipulation
+- hiro-robot
+- lerobot
+- goal-conditioned
+- sensor-diffusion
+datasets:
+- roboset_20260112_225816
+- roboset_20260113_001336
+---
+
+# Proximity Sensor Goal-Conditioned Diffusion Policy
+
+## Model Description
+
+A goal-conditioned Diffusion Policy trained on proximity sensor datasets. The model predicts joint positions (next positions along trajectory) conditioned on the current observation (joint positions, table camera image, encoded proximity sensor data) and a goal cartesian position.
+
+**Key Features:**
+- Uses 37 proximity sensors (8x8 depth maps) encoded to 128-dim latent via pretrained autoencoder
+- Visual input from table camera (480x640 RGB)
+- Goal-conditioned for reaching target cartesian positions
+- Predicts 16-step action horizon
+
+## Model Architecture
+
+- **Policy Type**: Diffusion Policy
+- **Framework**: LeRobot
+- **Horizon**: 16 steps
+- **Observation Steps**: 1 step (single timestep)
+- **Action Steps**: 8 steps (each covers 2 timesteps)
+- **Total Parameters**: ~261M
+
+## Inputs
+
+- **`observation.state`**: Shape `(batch, 1, 7)` - Joint positions (7 DOF arm)
+- **`observation.goal`**: Shape `(batch, 1, 3)` - Goal cartesian position (X, Y, Z)
+- **`observation.images.table_camera`**: Shape `(batch, 1, 3, 480, 640)` - Table camera RGB images
+- **`observation.proximity`**: Shape `(batch, 1, 128)` - Encoded proximity sensor latent (37 sensors → 128-dim via pretrained encoder)
+
+## Outputs
+
+- **`action`**: Shape `(batch, 16, 7)` - Joint positions (7 DOF) for 16-step horizon (next positions along trajectory)
+
+**Note**: The model outputs a full 16-step horizon. Use `select_action()` to get the first step `(batch, 7)`, or `predict_action_chunk()` to get the full horizon `(batch, 16, 7)`.
+
+## Normalization
+
+### Input Normalization
+
+**Images** (`observation.images.table_camera`):
+- Normalize from `[0, 255]` to `[0, 1]` by dividing by `255.0`
+- Then apply mean-std normalization using dataset statistics (handled by preprocessor)
+
+**State** (`observation.state`):
+- Apply min-max normalization: `(state - min) / (max - min)` using dataset statistics (handled by preprocessor)
+
+**Goal** (`observation.goal`):
+- Apply min-max normalization: `(goal - min) / (max - min)` using dataset statistics (handled by preprocessor)
+
+**Proximity** (`observation.proximity`):
+- Encoded via pretrained ProximityAutoencoder (frozen encoder)
+- 37 sensors × (8×8 depth maps) → 128-dim latent
+- Apply min-max normalization using dataset statistics (handled by preprocessor)
+
+### Output Unnormalization
+
+**Actions** (`action`):
+- Apply inverse min-max normalization: `action * (max - min) + min` using dataset statistics (handled by postprocessor)
+- **Note**: Actions are joint positions (not velocities) - these are the next positions the robot should move to along the trajectory
+
+## Usage
+
+```python
+from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy
+from lerobot.policies.factory import make_pre_post_processors
+
+# Load model
+policy = DiffusionPolicy.from_pretrained("calebescobedo/sensor-diffusion-policy-topdown-camera")
+
+# Load preprocessor and postprocessor from the same repo
+preprocessor, postprocessor = make_pre_post_processors(
+    policy_cfg=policy.config,
+    pretrained_path="calebescobedo/sensor-diffusion-policy-topdown-camera"
+)
+
+# Prepare inputs
+batch = {
+    'observation.state': state_tensor,  # (batch, 1, 7) - raw joint positions
+    'observation.goal': goal_tensor,  # (batch, 1, 3) - raw goal xyz
+    'observation.images.table_camera': table_img,  # (batch, 1, 3, 480, 640) - uint8 [0,255] or float [0,1]
+    'observation.proximity': proximity_latent,  # (batch, 1, 128) - encoded proximity sensor latent
+}
+
+# Inference
+policy.eval()
+with torch.no_grad():
+    batch = preprocessor(batch)  # Normalizes inputs
+    actions = policy.select_action(batch)  # Returns normalized actions
+    actions = postprocessor(actions)  # Unnormalizes to raw joint positions
+```
+
+## Training Details
+
+- **Training**: Epoch-based (ensures all trajectories seen)
+- **Epochs**: 60
+- **Batch Size**: 64
+- **Optimizer**: Adam (LeRobot preset)
+- **Learning Rate**: From LeRobot optimizer preset
+- **Mixed Precision**: Enabled (AMP)
+- **Data Loading**: Optimized with persistent file handles (4 workers, prefetch=2)
+- **Data Augmentation**: 
+  - State noise: 30% probability, scale=0.005
+  - Action noise: 30% probability, scale=0.0005
+  - Goal noise: 30% probability, scale=[0.003, 0.005, 0.0005] (X, Y, Z)
+- **Datasets**: 
+  - roboset_20260117_014645 (20 H5 files, ~500 trajectories, ~17,000 sequences)
+
+## Proximity Sensor Encoding
+
+The proximity sensors are encoded using a pretrained autoencoder:
+- **Encoder**: 37 sensors × (8×8 depth maps) → 128-dim latent
+- **Architecture**: Per-sensor CNN (8×8 → 4×4 → 2×2) + Multi-head attention aggregation
+- **Training**: Separate pretraining on depth reconstruction (MSE loss: ~0.118)
+- **Status**: Encoder frozen during policy training (no gradients)
+
+## Dataset Notes
+
+- **37 proximity sensors** per timestep (depth_sensor_link1_sensor_0 through depth_sensor_link6_sensor_7)
+- Each sensor provides **8×8 depth maps** (`depth_to_camera`)
+- **Table camera RGB images** (480×640×3)
+- **7-DOF joint positions**
+- **Goal-conditioned trajectories**: Each trajectory has a unique goal (final cartesian position)
+- **Goal distribution**: 
+  - X: [-0.239, 0.294] meters
+  - Y: [-0.284, 0.317] meters
+  - Z: [0.364, 0.579] meters
+- **Total**: ~500 trajectories, ~17,000 sequences
+
+## License
+
+MIT License

config.json

@@ -0,0 +1,91 @@
+{
+    "type": "diffusion",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                7
+            ]
+        },
+        "observation.goal": {
+            "type": "STATE",
+            "shape": [
+                3
+            ]
+        },
+        "observation.images.top_down_camera": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.proximity": {
+            "type": "STATE",
+            "shape": [
+                128
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                7
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "push_to_hub": true,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "horizon": 16,
+    "n_action_steps": 8,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MIN_MAX",
+        "ACTION": "MIN_MAX"
+    },
+    "drop_n_last_frames": 7,
+    "vision_backbone": "resnet18",
+    "crop_shape": null,
+    "crop_is_random": true,
+    "pretrained_backbone_weights": null,
+    "use_group_norm": true,
+    "spatial_softmax_num_keypoints": 32,
+    "use_separate_rgb_encoder_per_camera": false,
+    "down_dims": [
+        512,
+        1024,
+        2048
+    ],
+    "kernel_size": 5,
+    "n_groups": 8,
+    "diffusion_step_embed_dim": 128,
+    "use_film_scale_modulation": true,
+    "noise_scheduler_type": "DDPM",
+    "num_train_timesteps": 100,
+    "beta_schedule": "squaredcos_cap_v2",
+    "beta_start": 0.0001,
+    "beta_end": 0.02,
+    "prediction_type": "epsilon",
+    "clip_sample": true,
+    "clip_sample_range": 1.0,
+    "num_inference_steps": null,
+    "do_mask_loss_for_padding": false,
+    "optimizer_lr": 0.0001,
+    "optimizer_betas": [
+        0.95,
+        0.999
+    ],
+    "optimizer_eps": 1e-08,
+    "optimizer_weight_decay": 1e-06,
+    "scheduler_name": "cosine",
+    "scheduler_warmup_steps": 500
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e87ec522642979d34a6a3596d1c4c45b321f4c6febbcc7aee11f277371937d0
+size 1043939492

policy_postprocessor.json

@@ -0,0 +1,32 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              7
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28463b3b064269f62d7f2a8795a066930dbc0d7b31d994ad868ad609435453b1
+size 3552

policy_preprocessor.json

@@ -0,0 +1,68 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              7
+            ]
+          },
+          "observation.goal": {
+            "type": "STATE",
+            "shape": [
+              3
+            ]
+          },
+          "observation.images.top_down_camera": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.proximity": {
+            "type": "STATE",
+            "shape": [
+              128
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              7
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MIN_MAX",
+          "ACTION": "MIN_MAX"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_3_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28463b3b064269f62d7f2a8795a066930dbc0d7b31d994ad868ad609435453b1
+size 3552