README.md

metadata

license: mit
tags:
  - robotics
  - imitation-learning
  - diffusion-policy
  - cloth-manipulation

DP3 floating-gripper garment policies — Hang_Trousers

Five 3D Diffusion Policy variants trained on floating-gripper garment-hanging demonstrations in DexGarmentLab (Hang_Trousers_FloatGrip, 195 demos, 12,516 transitions, Isaac Sim).

They differ only in the observation representation and the point encoder, which is what makes them useful as an ablation: what does ground-truth cloth mesh state, and true mesh connectivity, buy over a depth point cloud?

EMA weights only, ~0.98 GiB each. The source repo is currently private; everything needed to use these checkpoints is below.

The five variants

EE MAE is held-out end-effector position error against demonstration actions. It is not task success — no policy here has been evaluated in closed loop yet.

file	encoder	observation	edges	epoch	EE MAE
trousers_pn_depth.ckpt	PointNet	depth cloud 2048×3	—	550	6.25 mm
trousers_gnn_depth.ckpt	mesh-GNN	depth cloud	kNN	200	6.85 mm
trousers_pn_mesh.ckpt	PointNet	decimated GT mesh + pothook	—	900	6.94 mm
trousers_gnn_mesh.ckpt	mesh-GNN	GT mesh nodes	true garment faces	300	7.24 mm
trousers_gnn_knn.ckpt	mesh-GNN	GT mesh nodes	kNN	200	7.35 mm

How to read this table

• Down a column (same encoder) isolates the representation. Across a row isolates the encoder. Never use a row to make a claim about representations.
• gnn_mesh vs gnn_knn is the sharpest comparison: identical nodes, identical encoder, identical parameter count — the only difference is whether edges come from true mesh faces or from k-nearest-neighbours. The gap is 0.11 mm, i.e. nothing.
• Checkpoints were selected by lowest EE MAE, not by validation loss. The two disagree for every run, and selecting on val_loss would be up to 3.3 mm worse — it is diffusion denoising MSE averaged over noise levels, not a measure of sampled action quality.

Honest caveats

• All five are within ~1 mm of each other on a 4-episode validation split. Treat them as near-indistinguishable.
• Each figure is an argmin over 6–20 checkpoints on that same small split, so the numbers are optimistic.
• They are learning: an observation-free "hold current position" baseline scores 48.9 mm and constant-velocity 28.3 mm, so the policies beat trivial baselines by 4.5–8×.
• Encoder capacity, normalizer range and padding were each checked and ruled out as explanations for the mesh variants underperforming. The likely reason is that the depth cloud already contains ~75 % garment points plus the randomised target hook in one view-consistent frame, and this task's difficulty is not cloth-state estimation.

Usage

hf download albilo/dp3-floatgrip --local-dir checkpoints_pretrained

Needs torch, diffusers, hydra-core, omegaconf, dill, einops, termcolor, and the diffusion_policy_3d package from DexGarmentLab's IL_Baselines/Diffusion_Policy_3D on PYTHONPATH (the config in each checkpoint instantiates classes by dotted path).

import torch, dill, hydra
from omegaconf import OmegaConf
OmegaConf.register_new_resolver("eval", eval, replace=True)

p      = torch.load("trousers_pn_depth.ckpt", map_location="cpu", pickle_module=dill)
policy = hydra.utils.instantiate(p["cfg"].policy)
policy.load_state_dict(p["state_dicts"]["ema_model"])
policy.cuda().eval()

action = policy.predict_action(obs)["action"]      # (B, 4, 8)

Observation — n_obs_steps = 3, so a history of 3 frames, oldest first:

key	shape	notes
point_cloud	(B, 3, 2048, 3) float32	xyz only, metres, ground removed, FPS-downsampled
agent_pos	(B, 3, 8) float32	[left_xyz, right_xyz, grip_L, grip_R]

trousers_gnn_mesh additionally requires edges (B, 3, E, 2), n_nodes (B, 3) and n_edges (B, 3); without them it raises KeyError: 'edges'. The other four do not — the kNN variants build their graph from coordinates at runtime.

Action — (B, 4, 8): four future steps of [left_xyz, right_xyz, grip_L, grip_R], absolute positions in metres, not deltas. There is no orientation (the grippers are attachment spheres; the demos never recorded any). The normalizer travels inside the state_dict, so actions come back already in metres — do not rescale. Gripper bits are regressed continuously; threshold at 0.5. Execute all 4 actions before re-planning.

Timing — demonstrations were recorded every 10 physics steps with gripper motion interpolated at 0.01 m/step, so one action ≈ one recorded frame. Matching this matters: a different velocity regime is out of distribution.

Training

195 successful demonstrations, 12,516 transitions, one H100 per run. PointNet cells 1000 epochs @ batch 256 (1.4 h); mesh-GNN cells 300 epochs @ batch 64 (4.4 h, gradient checkpointing — the dense padded edge activations peak at 77 GiB without it). The mesh-GNN is a MeshGraphNets-style encoder → processor over the garment graph; mesh nodes are decimated to 1234–1535 with topology taken from the garment USDs and vertex correspondence preserved per frame.

manifest.json records which training epoch each file came from.