README.md

# Sensor Diffusion Policy - Epoch 220

Diffusion policy model trained on proximity sensor data with table camera images.

## Model Details

- **Model Type**: Diffusion Policy
- **Training Epochs**: 220/300
- **Horizon**: 16 steps
- **Observation Steps**: 1
- **Action Steps**: 8

## Inputs

The model expects the following inputs:

### 1. `observation.state` (STATE)
- **Shape**: `(batch, 1, 7)`
- **Description**: Joint positions for 7-DOF arm
- **Normalization**: Min-max normalized using dataset statistics

### 2. `observation.goal` (STATE)
- **Shape**: `(batch, 1, 3)`
- **Description**: Goal cartesian position (X, Y, Z in meters)
- **Normalization**: Min-max normalized using dataset statistics

### 3. `observation.images.table_camera` (VISUAL)
- **Shape**: `(batch, 1, 3, 480, 640)`
- **Description**: RGB images from table camera
- **Normalization**: Mean-std normalized (normalized to [0, 1] then mean-std)

### 4. `observation.proximity` (STATE)
- **Shape**: `(batch, 1, 128)`
- **Description**: Encoded proximity sensor latent (37 sensors × 8×8 depth maps → 128-dim)
- **Normalization**: Min-max normalized using dataset statistics
- **Note**: Requires pretrained ProximityAutoencoder encoder

## Outputs

### `action` (ACTION)
- **Shape**: `(batch, 7)`
- **Description**: Joint positions (7-DOF) for the next timestep
- **Type**: Joint positions (not velocities) - these are the next positions the robot should move to
- **Normalization**: Output is unnormalized (raw joint positions in radians)

## Usage

```python
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy
from lerobot.policies.factory import make_pre_post_processors
import torch

# Load model and processors
repo_id = "calebescobedo/sensor-diffusion-policy-table-camera-epoch220"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

policy = DiffusionPolicy.from_pretrained(repo_id)
policy.eval()
policy.to(device)

preprocessor, postprocessor = make_pre_post_processors(
    policy_cfg=policy.config,
    pretrained_path=repo_id
)

# Prepare input batch
# Note: You need to encode proximity sensors using the ProximityAutoencoder first
batch = {
    "observation.state": torch.tensor([...]),  # Shape: (batch, 1, 7)
    "observation.goal": torch.tensor([...]),    # Shape: (batch, 1, 3)
    "observation.images.table_camera": torch.tensor([...]),  # Shape: (batch, 1, 3, 480, 640)
    "observation.proximity": torch.tensor([...]),  # Shape: (batch, 1, 128) - encoded
}

# Predict action
with torch.no_grad():
    batch_processed = preprocessor(batch)  # Normalizes inputs
    actions = policy.select_action(batch_processed)  # Returns normalized actions
    actions = postprocessor(actions)  # Unnormalizes to raw joint positions

# actions shape: (batch, 7) - joint positions in radians
```

## Proximity Sensor Encoding

The proximity sensors must be encoded before use. You need to load the ProximityAutoencoder:

```python
from architectures.proximity_autoencoder import ProximityAutoencoder
import torch

# Load proximity encoder
encoder_path = "path/to/proximity_autoencoder.pth"
ae_model = ProximityAutoencoder(num_sensors=37, depth_channels=1, latent_dim=128, use_attention=True)
ae_model.load_state_dict(torch.load(encoder_path, map_location='cpu'))
proximity_encoder = ae_model.encoder
proximity_encoder.eval()

# Encode proximity sensors (37 sensors × 8×8 depth maps)
# raw_proximity shape: (batch, 37, 8, 8)
encoded_proximity = proximity_encoder(raw_proximity)  # Shape: (batch, 128)
```

## Dataset Statistics

Dataset statistics are included in `config.json` under the `dataset_stats` key. These are used for normalization/unnormalization and were computed from the training dataset:
- `/home/caleb/datasets/sensor/roboset_20260117_014645/*.h5` (20 files, ~500 trajectories)

## Training Details

- **Dataset**: Sensor dataset with proximity sensors and table camera
- **Training Loss**: ~0.003-0.004 (at epoch 220)
- **Optimizer**: Adam (LeRobot preset)
- **Learning Rate**: From LeRobot optimizer preset
- **Mixed Precision**: Enabled (AMP)
- **Data Augmentation**: State noise (30% prob, scale=0.005), Action noise (30% prob, scale=0.0005), Goal noise (30% prob)

## Model Architecture

- **Vision Backbone**: ResNet18
- **Diffusion Steps**: 100
- **Noise Scheduler**: DDPM with squaredcos_cap_v2 beta schedule
- **Total Parameters**: ~261M

## Citation

If you use this model, please cite:
```bibtex
@misc{sensor-diffusion-policy-epoch220,
  author = {Caleb Escobedo},
  title = {Sensor Diffusion Policy - Epoch 220},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/calebescobedo/sensor-diffusion-policy-table-camera-epoch220}}
}
```