dpe and rpe checkpoints (N, seed 42, epoch 75) + training configs

README.md

@@ -0,0 +1,59 @@
+---
+tags:
+- robotics
+- diffusion-policy
+- dexterous-manipulation
+- bimanual
+- pose-estimation
+---
+
+# DP-T perceived-pose checkpoints (bimanual_assembly)
+
+These are pointflow diffusion policies (variant **N**: surface-normal point tokens, 75 epochs, training seed 42). They are trained on object poses from **our own perception** instead of the simulator's, for the DexJoCo `bimanual_assembly` task.
+
+Ground truth enters only as the action labels and the pointflow target. At evaluation the same perception runs live once per replan.
+
+| Folder | Training input (`obj_pose`) | Eval input | Success (120 paired episodes, seeds 0–3 × 30) |
+|---|---|---|---|
+| `dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42` | **dpe**: RGB tracker + depth tracker on **perfect** (simulator) depth, every step | same, live | **47.5 %** (Oracle with true poses: 51.7 %, p = 0.64) |
+| `dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42` | **rpe**: the same run's RGB tracker only, every step | same, live (no depth) | 28.3 % |
+
+Each folder mirrors `outputs/dp/<experiment>/` of the code:
+- `checkpoints/latest.ckpt`: epoch 75, about 143 MB;
+- `.hydra/config.yaml` and `.hydra/overrides.yaml`: the exact training config;
+- `logs.json.txt`: the training curve.
+
+Code, the full pipeline and a porting guide live in the DP-T repository, branch `cadfree-pointflow`:
+- [`docs/PERFECT_DEPTH_PIPELINE.md`](https://github.com/unichk/DP-T/blob/cadfree-pointflow/docs/PERFECT_DEPTH_PIPELINE.md)
+- [`slurm/main/README.md`](https://github.com/unichk/DP-T/blob/cadfree-pointflow/slurm/main/README.md)
+
+The training data is in the dataset repo [`b12902136/DP-T-perceived-pose-data`](https://huggingface.co/datasets/b12902136/DP-T-perceived-pose-data).
+
+## Use
+
+```bash
+# 1. the checkpoints, into the code's outputs/dp/
+hf download b12902136/DP-T-perceived-pose --local-dir outputs/dp
+#    or just one:  --include 'dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42/*'
+
+# 2. the perception's object models and the policy's object cloud (in the code repo, 4 MB)
+bash assets/perceived_pose/install.sh          # -> $VLA_ROOT/DP-T/outputs (else ./outputs)
+
+# 3. evaluate with live perception (1 GPU per seed; ~0.7 GPU-h per 30 episodes)
+PV=dpe VARIANT=N POSE=perceived SEED=42 EPISODES=30 RUNS=6 sbatch --array=0-3 slurm/main/eval.sbatch
+PV=rpe VARIANT=N POSE=perceived SEED=42 EPISODES=30 RUNS=6 sbatch --array=0-3 slurm/main/eval.sbatch
+```
+
+`eval.sbatch` finds the checkpoint at `outputs/dp/<experiment>/checkpoints/latest.ckpt`. `PV` sets the live perception, which must match the training data:
+- **dpe:** `--perceived-depth perfect --perceived-depth-fix --perceived-depth-model <shared_model_perfect_fused.npz> --perceived-canon --perceived-peg-spin hand --perceived-sam3-tf32`
+- **rpe:** the same without the three depth flags.
+
+With plain `eval_dp.py`, pass these flags yourself (see `slurm/main/common.sh`).
+
+The checkpoint config records the training machine's absolute path of the object cloud (`policy.cloud_override`). `eval_dp.py` (commit c3927e6 or later) maps it onto your `outputs/` when that path does not exist; you can also pass `--policy-cloud-file <cloud.npz>`.
+
+## Caveats
+
+- **Perfect depth.** dpe's depth is the simulator's own, noise-free, so it is an upper bound for an RGB-D sensor, not a real-camera result.
+- **Task-specific.** The perception is assembly-specific: two objects (peg and tray), with their SAM 3 prompts, meshes and canonical rotations. Another task needs its own models (see the porting checklist in the guide).
+- **Different training data.** These are not the GT-pose checkpoints of the main DP-T release.

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42/.hydra/config.yaml

@@ -0,0 +1,199 @@
+name: train_dpt_decoupled_pointflow
+_target_: dexjoco_dp.workspace.train_diffusion_transformer_image_workspace.TrainDiffusionTransformerImageWorkspace
+task_name: ${task.name}
+shape_meta: ${task.shape_meta}
+exp_name: dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42
+horizon: 16
+n_obs_steps: 2
+n_action_steps: 8
+n_latency_steps: 0
+dataset_obs_steps: ${n_obs_steps}
+past_action_visible: false
+keypoint_visible_rate: 1.0
+obs_as_cond: true
+policy:
+  _target_: dexjoco_dp.policy.decoupled_pointflow_policy.DecoupledPointflowPolicy
+  flow_stats: ${task.flow_stats}
+  cloud_file: ${oc.select:task.cloud_file,null}
+  aux_weight: 0.5
+  detach_flow: false
+  tok_dropout: 0.1
+  jitter_sigma: 0.0
+  tok_dim: 96
+  attn_log_every: 500
+  hand_kp: false
+  normals: true
+  contact_centres: 0
+  hand_kp_feat: false
+  hand_local: false
+  hand_local_mem: true
+  hand_local_empty: false
+  hand_tok_dropout: 0.0
+  hand_cloud: false
+  hand_flow_stats: ${oc.select:task.hand_flow_stats,null}
+  local_encoder: max
+  router: false
+  towers_obj_pose: true
+  joint_flow: false
+  flow_weight: 1.0
+  fourier_local: 0
+  fourier_relpos: 0
+  msg_radius: 0.0
+  msg_k: 32
+  point_drop: 0.0
+  action_rope: false
+  freeze_gates: false
+  shape_meta: ${shape_meta}
+  noise_scheduler:
+    _target_: diffusers.schedulers.scheduling_ddpm.DDPMScheduler
+    num_train_timesteps: 100
+    beta_start: 0.0001
+    beta_end: 0.02
+    beta_schedule: squaredcos_cap_v2
+    variance_type: fixed_small
+    clip_sample: true
+    prediction_type: epsilon
+  horizon: ${horizon}
+  n_action_steps: ${eval:'${n_action_steps}+${n_latency_steps}'}
+  n_obs_steps: ${n_obs_steps}
+  num_inference_steps: 100
+  sampler_steps: 16
+  n_layer: 8
+  n_cond_layers: 0
+  n_head: 4
+  n_emb: 176
+  p_drop_emb: 0.0
+  p_drop_attn: 0.3
+  causal_attn: true
+  time_as_cond: true
+  obs_as_cond: ${obs_as_cond}
+  cloud_override: /work/b12902136/DP-T/outputs/sam3d/perceived_v9/cloud.npz
+ema:
+  _target_: diffusion_policy.model.diffusion.ema_model.EMAModel
+  update_after_step: 0
+  inv_gamma: 1.0
+  power: 0.75
+  min_value: 0.0
+  max_value: 0.9999
+dataloader:
+  batch_size: 512
+  num_workers: 2
+  shuffle: true
+  pin_memory: true
+  persistent_workers: false
+val_dataloader:
+  batch_size: 512
+  num_workers: 1
+  shuffle: false
+  pin_memory: true
+  persistent_workers: false
+optimizer:
+  transformer_weight_decay: 0.001
+  obs_encoder_weight_decay: 1.0e-06
+  learning_rate: 0.000283
+  betas:
+  - 0.9
+  - 0.95
+training:
+  device: cuda:0
+  seed: 42
+  debug: false
+  resume: true
+  lr_scheduler: cosine
+  lr_warmup_steps: 1000
+  num_epochs: 1000
+  gradient_accumulate_every: 1
+  use_ema: true
+  rollout_every: 50
+  checkpoint_every: 25
+  val_every: 1
+  sample_every: 5
+  max_train_steps: null
+  max_val_steps: null
+  tqdm_interval_sec: 5.0
+  stop_after_epoch: 75
+logging:
+  project: dexjoco_dp
+  resume: true
+  mode: offline
+  name: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}
+  tags:
+  - ${name}
+  - ${task_name}
+  - ${exp_name}
+  id: null
+  group: null
+checkpoint:
+  topk:
+    monitor_key: val_loss
+    mode: min
+    k: 5
+    format_str: epoch={epoch:04d}-val_loss={val_loss:.4f}.ckpt
+  save_last_ckpt: true
+  save_last_snapshot: false
+multi_run:
+  run_dir: outputs/dp/${exp_name}
+  wandb_name_base: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}
+task:
+  name: dexjoco_assembly_lowdim_delta_pointflow_perceived
+  image_shape:
+  - 3
+  - 240
+  - 240
+  dexjoco_root: ${oc.env:DEXJOCO_ROOT,/work/unicorn1612/VLA/future/dexjoco}
+  perceived_version: dpe
+  dataset_path: ${task.dexjoco_root}/converted_datasets/zarr/rand_obj_objfut_pointflow_perceived_${task.perceived_version}/bimanual_assembly
+  flow_stats: ${task.dataset_path}/pointflow_stats.npz
+  eval:
+    config: ${task.dexjoco_root}/configs/rand_obj/bimanual_assembly.yaml
+    camera_keys:
+      camera_0: ego
+    rand_full: false
+  shape_meta:
+    obs:
+      state:
+        shape:
+        - 46
+        type: low_dim
+      obj_pose:
+        shape:
+        - 14
+        type: low_dim
+      obj_pose_fut:
+        shape:
+        - 56
+        type: low_dim
+    action:
+      shape:
+      - 44
+  env_runner: null
+  dataset:
+    _target_: dexjoco_dp.dataset.dexjoco_image_dataset.DexJoCoImageDataset
+    shape_meta:
+      obs:
+        state:
+          shape:
+          - 46
+          type: low_dim
+        obj_pose:
+          shape:
+          - 14
+          type: low_dim
+        obj_pose_fut:
+          shape:
+          - 56
+          type: low_dim
+      action:
+        shape:
+        - 44
+    dataset_path: ${task.dataset_path}
+    horizon: ${horizon}
+    pad_before: ${eval:'${n_obs_steps}-1+${n_latency_steps}'}
+    pad_after: ${eval:'${n_action_steps}-1'}
+    n_obs_steps: ${dataset_obs_steps}
+    n_latency_steps: ${n_latency_steps}
+    use_cache: true
+    seed: 42
+    val_ratio: 0.05
+    max_train_episodes: null
+    delta_action: true

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42/.hydra/overrides.yaml

@@ -0,0 +1,16 @@
+- policy.normals=True
+- policy.contact_centres=0
+- optimizer.learning_rate=2.83e-4
+- training.num_epochs=1000
+- training.lr_warmup_steps=1000
+- training.checkpoint_every=25
+- +training.stop_after_epoch=75
+- dataloader.num_workers=2
+- training.seed=42
+- dataloader.batch_size=512
+- val_dataloader.batch_size=512
+- val_dataloader.num_workers=1
+- task=dexjoco_assembly_lowdim_delta_pointflow_perceived
+- task.perceived_version=dpe
+- +policy.cloud_override=/work/b12902136/DP-T/outputs/sam3d/perceived_v9/cloud.npz
+- exp_name=dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_dpe_s42/checkpoints/latest.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29e682fa6b1b036eccc3ae1a28fcc6cdef3a9305bf10ba14e74d13023d63aa60
+size 142619090

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42/.hydra/config.yaml

@@ -0,0 +1,199 @@
+name: train_dpt_decoupled_pointflow
+_target_: dexjoco_dp.workspace.train_diffusion_transformer_image_workspace.TrainDiffusionTransformerImageWorkspace
+task_name: ${task.name}
+shape_meta: ${task.shape_meta}
+exp_name: dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42
+horizon: 16
+n_obs_steps: 2
+n_action_steps: 8
+n_latency_steps: 0
+dataset_obs_steps: ${n_obs_steps}
+past_action_visible: false
+keypoint_visible_rate: 1.0
+obs_as_cond: true
+policy:
+  _target_: dexjoco_dp.policy.decoupled_pointflow_policy.DecoupledPointflowPolicy
+  flow_stats: ${task.flow_stats}
+  cloud_file: ${oc.select:task.cloud_file,null}
+  aux_weight: 0.5
+  detach_flow: false
+  tok_dropout: 0.1
+  jitter_sigma: 0.0
+  tok_dim: 96
+  attn_log_every: 500
+  hand_kp: false
+  normals: true
+  contact_centres: 0
+  hand_kp_feat: false
+  hand_local: false
+  hand_local_mem: true
+  hand_local_empty: false
+  hand_tok_dropout: 0.0
+  hand_cloud: false
+  hand_flow_stats: ${oc.select:task.hand_flow_stats,null}
+  local_encoder: max
+  router: false
+  towers_obj_pose: true
+  joint_flow: false
+  flow_weight: 1.0
+  fourier_local: 0
+  fourier_relpos: 0
+  msg_radius: 0.0
+  msg_k: 32
+  point_drop: 0.0
+  action_rope: false
+  freeze_gates: false
+  shape_meta: ${shape_meta}
+  noise_scheduler:
+    _target_: diffusers.schedulers.scheduling_ddpm.DDPMScheduler
+    num_train_timesteps: 100
+    beta_start: 0.0001
+    beta_end: 0.02
+    beta_schedule: squaredcos_cap_v2
+    variance_type: fixed_small
+    clip_sample: true
+    prediction_type: epsilon
+  horizon: ${horizon}
+  n_action_steps: ${eval:'${n_action_steps}+${n_latency_steps}'}
+  n_obs_steps: ${n_obs_steps}
+  num_inference_steps: 100
+  sampler_steps: 16
+  n_layer: 8
+  n_cond_layers: 0
+  n_head: 4
+  n_emb: 176
+  p_drop_emb: 0.0
+  p_drop_attn: 0.3
+  causal_attn: true
+  time_as_cond: true
+  obs_as_cond: ${obs_as_cond}
+  cloud_override: /work/b12902136/DP-T/outputs/sam3d/perceived_v9/cloud.npz
+ema:
+  _target_: diffusion_policy.model.diffusion.ema_model.EMAModel
+  update_after_step: 0
+  inv_gamma: 1.0
+  power: 0.75
+  min_value: 0.0
+  max_value: 0.9999
+dataloader:
+  batch_size: 512
+  num_workers: 2
+  shuffle: true
+  pin_memory: true
+  persistent_workers: false
+val_dataloader:
+  batch_size: 512
+  num_workers: 1
+  shuffle: false
+  pin_memory: true
+  persistent_workers: false
+optimizer:
+  transformer_weight_decay: 0.001
+  obs_encoder_weight_decay: 1.0e-06
+  learning_rate: 0.000283
+  betas:
+  - 0.9
+  - 0.95
+training:
+  device: cuda:0
+  seed: 42
+  debug: false
+  resume: true
+  lr_scheduler: cosine
+  lr_warmup_steps: 1000
+  num_epochs: 1000
+  gradient_accumulate_every: 1
+  use_ema: true
+  rollout_every: 50
+  checkpoint_every: 25
+  val_every: 1
+  sample_every: 5
+  max_train_steps: null
+  max_val_steps: null
+  tqdm_interval_sec: 5.0
+  stop_after_epoch: 75
+logging:
+  project: dexjoco_dp
+  resume: true
+  mode: offline
+  name: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}
+  tags:
+  - ${name}
+  - ${task_name}
+  - ${exp_name}
+  id: null
+  group: null
+checkpoint:
+  topk:
+    monitor_key: val_loss
+    mode: min
+    k: 5
+    format_str: epoch={epoch:04d}-val_loss={val_loss:.4f}.ckpt
+  save_last_ckpt: true
+  save_last_snapshot: false
+multi_run:
+  run_dir: outputs/dp/${exp_name}
+  wandb_name_base: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}
+task:
+  name: dexjoco_assembly_lowdim_delta_pointflow_perceived
+  image_shape:
+  - 3
+  - 240
+  - 240
+  dexjoco_root: ${oc.env:DEXJOCO_ROOT,/work/unicorn1612/VLA/future/dexjoco}
+  perceived_version: rpe
+  dataset_path: ${task.dexjoco_root}/converted_datasets/zarr/rand_obj_objfut_pointflow_perceived_${task.perceived_version}/bimanual_assembly
+  flow_stats: ${task.dataset_path}/pointflow_stats.npz
+  eval:
+    config: ${task.dexjoco_root}/configs/rand_obj/bimanual_assembly.yaml
+    camera_keys:
+      camera_0: ego
+    rand_full: false
+  shape_meta:
+    obs:
+      state:
+        shape:
+        - 46
+        type: low_dim
+      obj_pose:
+        shape:
+        - 14
+        type: low_dim
+      obj_pose_fut:
+        shape:
+        - 56
+        type: low_dim
+    action:
+      shape:
+      - 44
+  env_runner: null
+  dataset:
+    _target_: dexjoco_dp.dataset.dexjoco_image_dataset.DexJoCoImageDataset
+    shape_meta:
+      obs:
+        state:
+          shape:
+          - 46
+          type: low_dim
+        obj_pose:
+          shape:
+          - 14
+          type: low_dim
+        obj_pose_fut:
+          shape:
+          - 56
+          type: low_dim
+      action:
+        shape:
+        - 44
+    dataset_path: ${task.dataset_path}
+    horizon: ${horizon}
+    pad_before: ${eval:'${n_obs_steps}-1+${n_latency_steps}'}
+    pad_after: ${eval:'${n_action_steps}-1'}
+    n_obs_steps: ${dataset_obs_steps}
+    n_latency_steps: ${n_latency_steps}
+    use_cache: true
+    seed: 42
+    val_ratio: 0.05
+    max_train_episodes: null
+    delta_action: true

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42/.hydra/overrides.yaml

@@ -0,0 +1,16 @@
+- policy.normals=True
+- policy.contact_centres=0
+- optimizer.learning_rate=2.83e-4
+- training.num_epochs=1000
+- training.lr_warmup_steps=1000
+- training.checkpoint_every=25
+- +training.stop_after_epoch=75
+- dataloader.num_workers=2
+- training.seed=42
+- dataloader.batch_size=512
+- val_dataloader.batch_size=512
+- val_dataloader.num_workers=1
+- task=dexjoco_assembly_lowdim_delta_pointflow_perceived
+- task.perceived_version=rpe
+- +policy.cloud_override=/work/b12902136/DP-T/outputs/sam3d/perceived_v9/cloud.npz
+- exp_name=dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42

dpt_assembly_rand_obj_decoupled_pointflow_main_N_perc_rpe_s42/checkpoints/latest.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86d344e81da8fbbf8648087b1eb9fba7bad41fc4c3bde27655e17cb87063171f
+size 142619090