Ngseo/rh56f1_diffusion_dinov2s_flowmatch_multicam_dr

rh56f1 Diffusion Policy — FlowMatch + multicam (d405+zivid) + DINOv2-S + DR

LeRobot Diffusion Policy trained on the CrazyMoment/teleop_recorded_rh56f1_hookonly teleop dataset (HDR35_20 + RH56F1_R hand, "remove hook ring from chassis" task).

Model summary

Algorithm	Diffusion Policy (1D UNet) with rectified-flow loss
Noise scheduler	FlowMatch (num_inference_steps=1, Euler ODE)
Vision backbone	facebook/dinov2-small (frozen)
Cameras	d405 (240×320) + zivid (240×320, downsampled from 1050×1458)
State dim	141
Action dim	12
Horizon / n_obs_steps / n_action_steps	16 / 2 / 8
Training steps	200,000 (batch 64, ~223 epochs over 57,388 frames)
Image augmentations	lerobot image_transforms with domain randomization (p=0.5, max_num=3)
Optimizer	AdamW, lr=1e-4 cosine (warmup 500), wd=1e-6, β=(0.95, 0.999)
Final train loss	0.013
Hardware	NVIDIA A100 80GB · 24h 43m

Files

File	Purpose
model.safetensors	Policy weights
config.json	DiffusionConfig
train_config.json	Full training config snapshot
policy_preprocessor.json + .safetensors	Normalizer pipeline (state/action MIN_MAX, visual IDENTITY)
policy_postprocessor.json + .safetensors	Action unnormalizer pipeline

Inference

Install LeRobot from source (the FlowMatch scheduler lives in the diffusion policy):

git clone https://github.com/huggingface/lerobot
cd lerobot && pip install -e .
pip install transformers

Single-action inference from observations

import torch
from lerobot.configs.policies import PreTrainedConfig
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy
from lerobot.policies.factory import make_pre_post_processors

REPO = "Ngseo/rh56f1_diffusion_dinov2s_flowmatch_multicam_dr"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# 1) Load config + policy + pre/post processors from the Hub
cfg = PreTrainedConfig.from_pretrained(REPO)
cfg.device = DEVICE
policy = DiffusionPolicy.from_pretrained(REPO, config=cfg).to(DEVICE).eval()
preprocessor, postprocessor = make_pre_post_processors(cfg, pretrained_path=REPO)

# 2) Build an observation. Images are CHW float32 in [0,1] at 240×320.
def make_obs():
    return {
        "observation.state":         torch.zeros(141, dtype=torch.float32),
        "observation.images.d405":   torch.zeros(3, 240, 320, dtype=torch.float32),
        "observation.images.zivid":  torch.zeros(3, 240, 320, dtype=torch.float32),
    }

def to_batch(sample):
    """Add a batch dim. The preprocessor moves to device + normalizes."""
    return {k: (v.unsqueeze(0) if isinstance(v, torch.Tensor) else v) for k, v in sample.items()}

# 3) Roll out n_action_steps actions (default 8) without re-running the diffusion head.
#    The policy caches an action chunk and emits one action per call to select_action.
policy.reset()
for t in range(cfg.n_action_steps):
    obs = make_obs()                              # ← replace with real cameras + state
    batch = preprocessor(to_batch(obs))
    with torch.no_grad():
        action_norm = policy.select_action(batch) # (1, 12) — normalized
    action = postprocessor(action_norm.squeeze(0)).cpu().numpy()  # (12,) — real units
    print(f"t={t}  action={action}")

Key shape requirements:

• observation.state : (141,) float32
• observation.images.d405 / observation.images.zivid : (3, 240, 320) float32 in [0, 1]
• The preprocessor handles batch-dim, device transfer, and MIN-MAX normalization for state/action.
• The postprocessor reverses action MIN-MAX normalization on the CPU.

Replay against the recorded dataset

If you have the LeRobot-format dataset locally (after running examples/port_datasets/port_hookonly.py --cameras d405,zivid):

python examples/port_datasets/inference_diffusion_hookonly.py \
    --checkpoint <local-or-hub-path>/Ngseo--rh56f1_diffusion_dinov2s_flowmatch_multicam_dr \
    --dataset-root /path/to/lerobot_data/rh56f1_hookonly_multicam \
    --frame-index 100

The script streams observations frame-by-frame, calls policy.select_action, and prints ground-truth vs. predicted actions for visual comparison.

Reproducing training

The dataset must first be ported from the raw HDF5 release to LeRobot v3 format (d405 + zivid, 240×320, 60 fps):

python examples/port_datasets/port_hookonly.py \
    --src   /path/to/teleop_recorded_rh56f1_hookonly \
    --out   /path/to/lerobot_data/rh56f1_hookonly_multicam \
    --robot rh56f1 \
    --cameras d405,zivid \
    --repo-id local/rh56f1_hookonly_multicam \
    --streaming-encoding         # ~6× faster than the default

Then train:

lerobot-train \
    --dataset.repo_id=local/rh56f1_hookonly_multicam \
    --dataset.root=/path/to/lerobot_data/rh56f1_hookonly_multicam \
    --dataset.image_transforms.enable=true \
    --dataset.image_transforms.p_apply=0.5 \
    --dataset.image_transforms.max_num_transforms=3 \
    --dataset.image_transforms.domain_randomization=true \
    --policy.type=diffusion \
    --policy.vision_backbone=dinov2 \
    --policy.dinov2_model_name=facebook/dinov2-small \
    --policy.freeze_vision_backbone=true \
    --policy.spatial_softmax_num_keypoints=64 \
    --policy.noise_scheduler_type=FlowMatch \
    --policy.num_inference_steps=1 \
    --policy.device=cuda \
    --policy.push_to_hub=false \
    --output_dir=outputs/train/rh56f1_diffusion_dinov2s_flowmatch_multicam_dr \
    --job_name=rh56f1_diffusion_dinov2s_flowmatch_multicam_dr \
    --batch_size=64 --steps=200000 --eval_freq=0

Intended use & limitations

This checkpoint was trained on a single teleop task ("remove hook ring from chassis") with one robot embodiment (HDR35_20 + RH56F1_R right hand) using only 202 episodes (57k frames). It is intended for:

• Reproducing the FlowMatch + DINOv2-S + multicam + DR ablation in the rh56f1_* matrix.
• Sim-to-real / real-deployment experiments on the same hardware.

It will not generalize to other tasks, hand kinematics, or camera layouts without finetuning.