Add files using upload-large-folder tool

adroit_door-dp3-1218_seed0/train.log

@@ -0,0 +1,2 @@
+[2024-12-19 02:17:59,866][diffusion_policy_3d.model.diffusion.conditional_unet1d][INFO] - number of parameters: 2.551605e+08
+[2024-12-19 02:21:36,315][diffusion_policy_3d.model.diffusion.conditional_unet1d][INFO] - number of parameters: 2.551605e+08

adroit_door-dp3-1218_seed0/wandb/debug.log

@@ -0,0 +1,24 @@
+2024-12-19 02:21:44,367 INFO    MainThread:4155756 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_setup.py:_flush():68] Configure stats pid to 4155756
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_door-dp3-1218_seed0/wandb/run-20241219_022144-lubq05rh/logs/debug.log
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_door-dp3-1218_seed0/wandb/run-20241219_022144-lubq05rh/logs/debug-internal.log
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:init():644] calling init triggers
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'adroit_door', 'task_name': 'door', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [24], 'type': 'low_dim'}}, 'action': {'shape': [28]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.adroit_runner.AdroitRunner', 'eval_episodes': 20, 'max_steps': 300, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'task_name': 'door', 'render_size': 84, 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.adroit_dataset.AdroitDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/adroit_door_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'adroit_door', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [24], 'type': 'low_dim'}}, 'action': {'shape': [28]}}, 'exp_name': 'adroit_door-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [24], 'type': 'low_dim'}}, 'action': {'shape': [28]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'adroit_door-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.19/02.21.41_train_dp3_adroit_door', 'wandb_name_base': '2024.12.19-02.21.41_train_dp3_adroit_door'}}
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:init():680] starting backend
+2024-12-19 02:21:44,368 INFO    MainThread:4155756 [wandb_init.py:init():684] sending inform_init request
+2024-12-19 02:21:44,399 INFO    MainThread:4155756 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-19 02:21:44,399 INFO    MainThread:4155756 [wandb_init.py:init():697] backend started and connected
+2024-12-19 02:21:44,405 INFO    MainThread:4155756 [wandb_init.py:init():790] updated telemetry
+2024-12-19 02:21:44,568 INFO    MainThread:4155756 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-19 02:21:47,254 INFO    MainThread:4155756 [wandb_init.py:init():874] starting run threads in backend
+2024-12-19 02:21:47,372 INFO    MainThread:4155756 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-19 02:21:47,372 INFO    MainThread:4155756 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-19 02:21:47,372 INFO    MainThread:4155756 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-19 02:21:47,372 INFO    MainThread:4155756 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-19 02:21:47,375 INFO    MainThread:4155756 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-19 02:21:47,375 INFO    MainThread:4155756 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_door-dp3-1218_seed0'}
+2024-12-19 03:43:48,464 WARNING MsgRouterThr:4155756 [router.py:message_loop():75] message_loop has been closed

adroit_door-dp3-1218_seed0/wandb/run-20241219_022144-lubq05rh/files/config.yaml

@@ -0,0 +1,240 @@
+_wandb:
+    value:
+        cli_version: 0.19.1
+        m: []
+        python_version: 3.8.20
+        t:
+            "1":
+                - 1
+                - 5
+                - 41
+                - 49
+                - 50
+                - 53
+                - 55
+                - 83
+            "2":
+                - 1
+                - 5
+                - 41
+                - 49
+                - 50
+                - 53
+                - 55
+                - 83
+            "3":
+                - 13
+                - 15
+                - 16
+                - 23
+                - 55
+                - 61
+            "4": 3.8.20
+            "5": 0.19.1
+            "8":
+                - 5
+            "12": 0.19.1
+            "13": linux-x86_64
+checkpoint:
+    value:
+        save_ckpt: true
+        save_last_ckpt: true
+        save_last_snapshot: false
+        topk:
+            format_str: epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt
+            k: 1
+            mode: max
+            monitor_key: test_mean_score
+dataloader:
+    value:
+        batch_size: 128
+        num_workers: 8
+        persistent_workers: false
+        pin_memory: true
+        shuffle: true
+dataset_obs_steps:
+    value: 2
+ema:
+    value:
+        _target_: diffusion_policy_3d.model.diffusion.ema_model.EMAModel
+        inv_gamma: 1
+        max_value: 0.9999
+        min_value: 0
+        power: 0.75
+        update_after_step: 0
+exp_name:
+    value: adroit_door-dp3-1218
+horizon:
+    value: 4
+keypoint_visible_rate:
+    value: 1
+logging:
+    value:
+        group: adroit_door-dp3-1218
+        id: null
+        mode: online
+        name: "0"
+        project: dp3
+        resume: true
+        tags:
+            - dp3
+multi_run:
+    value:
+        run_dir: data/outputs/2024.12.19/02.21.41_train_dp3_adroit_door
+        wandb_name_base: 2024.12.19-02.21.41_train_dp3_adroit_door
+n_action_steps:
+    value: 3
+n_latency_steps:
+    value: 0
+n_obs_steps:
+    value: 2
+name:
+    value: train_dp3
+obs_as_global_cond:
+    value: true
+optimizer:
+    value:
+        _target_: torch.optim.AdamW
+        betas:
+            - 0.95
+            - 0.999
+        eps: 1e-08
+        lr: 0.0001
+        weight_decay: 1e-06
+output_dir:
+    value: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_door-dp3-1218_seed0
+policy:
+    value:
+        _target_: diffusion_policy_3d.policy.dp3.DP3
+        condition_type: film
+        crop_shape:
+            - 80
+            - 80
+        diffusion_step_embed_dim: 128
+        down_dims:
+            - 512
+            - 1024
+            - 2048
+        encoder_output_dim: 64
+        horizon: 4
+        kernel_size: 5
+        n_action_steps: 3
+        n_groups: 8
+        n_obs_steps: 2
+        noise_scheduler:
+            _target_: diffusers.schedulers.scheduling_ddim.DDIMScheduler
+            beta_end: 0.02
+            beta_schedule: squaredcos_cap_v2
+            beta_start: 0.0001
+            clip_sample: true
+            num_train_timesteps: 100
+            prediction_type: sample
+            set_alpha_to_one: true
+            steps_offset: 0
+        num_inference_steps: 10
+        obs_as_global_cond: true
+        pointcloud_encoder_cfg:
+            final_norm: layernorm
+            in_channels: 3
+            normal_channel: false
+            out_channels: 64
+            use_layernorm: true
+        pointnet_type: pointnet
+        shape_meta:
+            action:
+                shape:
+                    - 28
+            obs:
+                agent_pos:
+                    shape:
+                        - 24
+                    type: low_dim
+                point_cloud:
+                    shape:
+                        - 512
+                        - 3
+                    type: point_cloud
+        use_down_condition: true
+        use_mid_condition: true
+        use_pc_color: false
+        use_point_crop: true
+        use_up_condition: true
+shape_meta:
+    value:
+        action:
+            shape:
+                - 28
+        obs:
+            agent_pos:
+                shape:
+                    - 24
+                type: low_dim
+            point_cloud:
+                shape:
+                    - 512
+                    - 3
+                type: point_cloud
+task:
+    value:
+        dataset:
+            _target_: diffusion_policy_3d.dataset.adroit_dataset.AdroitDataset
+            horizon: 4
+            max_train_episodes: 90
+            pad_after: 2
+            pad_before: 1
+            seed: 42
+            val_ratio: 0.02
+            zarr_path: /data/haojun/datasets/3d-dp/adroit_door_expert.zarr
+        env_runner:
+            _target_: diffusion_policy_3d.env_runner.adroit_runner.AdroitRunner
+            eval_episodes: 20
+            fps: 10
+            max_steps: 300
+            n_action_steps: 3
+            n_obs_steps: 2
+            render_size: 84
+            task_name: door
+            use_point_crop: true
+        name: adroit_door
+        shape_meta:
+            action:
+                shape:
+                    - 28
+            obs:
+                agent_pos:
+                    shape:
+                        - 24
+                    type: low_dim
+                point_cloud:
+                    shape:
+                        - 512
+                        - 3
+                    type: point_cloud
+        task_name: door
+task_name:
+    value: adroit_door
+training:
+    value:
+        checkpoint_every: 200
+        debug: false
+        device: cuda:0
+        gradient_accumulate_every: 1
+        lr_scheduler: cosine
+        lr_warmup_steps: 500
+        max_train_steps: null
+        max_val_steps: null
+        num_epochs: 3000
+        resume: true
+        rollout_every: 200
+        sample_every: 5
+        seed: 0
+        tqdm_interval_sec: 1
+        use_ema: true
+        val_every: 1
+val_dataloader:
+    value:
+        batch_size: 128
+        num_workers: 8
+        persistent_workers: false
+        pin_memory: true
+        shuffle: false

adroit_door-dp3-1218_seed0/wandb/run-20241219_022144-lubq05rh/files/output.log

@@ -0,0 +1,16 @@
+                                                                                                                             
+test_mean_score: 0.0
+test_mean_score: 0.5
+test_mean_score: 0.65
+test_mean_score: 0.5
+test_mean_score: 0.45
+test_mean_score: 0.55
+test_mean_score: 0.35
+test_mean_score: 0.55
+test_mean_score: 0.45
+test_mean_score: 0.6
+test_mean_score: 0.4
+test_mean_score: 0.35
+test_mean_score: 0.25
+test_mean_score: 0.35
+test_mean_score: 0.575

adroit_door-dp3-1218_seed0/wandb/run-20241219_022144-lubq05rh/files/requirements.txt

@@ -0,0 +1,175 @@
+diffusion_policy_3d==0.0.0
+certifi==2024.12.14
+PyYAML==6.0.2
+dash-table==5.0.0
+dm_control==1.0.23
+einops==0.4.1
+ptyprocess==0.7.0
+jsonschema==4.23.0
+prompt_toolkit==3.0.48
+nbformat==5.10.4
+decorator==4.4.2
+importlib_metadata==8.5.0
+numcodecs==0.12.1
+requests==2.32.3
+ipywidgets==8.1.5
+urllib3==2.2.3
+platformdirs==4.3.6
+imageio-ffmpeg==0.5.1
+dash==2.18.2
+Cython==0.29.35
+mujoco-py==2.1.2.14
+mpmath==1.3.0
+typing_extensions==4.12.2
+dm-tree==0.1.8
+imageio==2.35.1
+dill==0.3.5.1
+joblib==1.4.2
+attrs==24.3.0
+torchvision==0.19.1+cu121
+kiwisolver==1.4.7
+numpy==1.23.5
+pillow==10.2.0
+nvidia-cuda-cupti-cu12==12.1.105
+pip==24.2
+llvmlite==0.39.1
+tenacity==9.0.0
+scipy==1.10.1
+charset-normalizer==3.4.0
+matplotlib==3.7.5
+jupyterlab_widgets==3.0.13
+patchelf==0.17.2.0
+wcwidth==0.2.13
+pexpect==4.9.0
+plotly==5.24.1
+ipdb==0.13.13
+referencing==0.35.1
+mujoco==3.2.3
+portalocker==3.0.0
+jedi==0.19.2
+Jinja2==3.1.3
+MarkupSafe==2.1.5
+nvidia-nvtx-cu12==12.1.105
+python-dateutil==2.9.0.post0
+packaging==24.2
+click==8.1.7
+pycparser==2.22
+blinker==1.8.2
+sympy==1.13.1
+Werkzeug==3.0.6
+six==1.17.0
+stack-data==0.6.3
+gpustat==1.1.1
+threadpoolctl==3.5.0
+traitlets==5.14.3
+annotated-types==0.7.0
+nvidia-nvjitlink-cu12==12.1.105
+zarr==2.12.0
+pandas==2.0.3
+wheel==0.44.0
+nvidia-nccl-cu12==2.20.5
+comm==0.2.2
+proglog==0.1.10
+parso==0.8.4
+open3d==0.18.0
+nvidia-cusparse-cu12==12.1.0.106
+antlr4-python3-runtime==4.9.3
+fvcore==0.1.5.post20221221
+natsort==8.4.0
+cloudpickle==3.1.0
+ConfigArgParse==1.7
+fonttools==4.55.3
+etils==1.3.0
+torchaudio==2.4.1+cu121
+pickleshare==0.7.5
+eval_type_backport==0.2.0
+executing==2.1.0
+addict==2.4.0
+protobuf==5.29.1
+setproctitle==1.3.4
+Flask==3.0.3
+filelock==3.13.1
+regex==2024.11.6
+PyOpenGL==3.1.7
+nvidia-curand-cu12==10.3.2.106
+matplotlib-inline==0.1.7
+transforms3d==0.4.2
+pydantic==2.10.3
+cffi==1.17.1
+rpds-py==0.20.1
+diffusers==0.11.1
+docker-pycreds==0.4.0
+pyparsing==3.1.4
+absl-py==2.1.0
+nvidia-cufft-cu12==11.0.2.54
+contourpy==1.1.1
+fastjsonschema==2.21.1
+pkgutil_resolve_name==1.3.10
+dash-html-components==2.0.0
+asciitree==0.3.3
+fasteners==0.19
+glfw==2.8.0
+omegaconf==2.3.0
+asttokens==3.0.0
+moviepy==1.0.3
+GitPython==3.1.43
+opencv-python==4.10.0.84
+zipp==3.20.2
+tabulate==0.9.0
+iopath==0.1.10
+labmaze==1.0.6
+numba==0.56.4
+nvidia-cuda-runtime-cu12==12.1.105
+dm-env==1.6
+cycler==0.12.1
+dash-core-components==2.0.0
+networkx==3.0
+Pygments==2.18.0
+retrying==1.3.4
+jsonschema-specifications==2023.12.1
+setuptools==59.5.0
+av==12.3.0
+itsdangerous==2.2.0
+wandb==0.19.1
+nest-asyncio==1.6.0
+triton==3.0.0
+nvidia-ml-py==12.560.30
+nvidia-cuda-nvrtc-cu12==12.1.105
+tomli==2.2.1
+tqdm==4.67.1
+termcolor==2.4.0
+hydra-core==1.2.0
+smmap==5.0.1
+ipython==8.12.3
+pyquaternion==0.9.9
+idna==3.10
+yacs==0.1.8
+tzdata==2024.2
+torch==2.4.1+cu121
+widgetsnbextension==4.0.13
+scikit-learn==1.3.2
+sapien==2.2.1
+nvidia-cublas-cu12==12.1.3.1
+huggingface-hub==0.21.0
+gitdb==4.0.11
+importlib_resources==6.4.5
+pydantic_core==2.27.1
+jupyter_core==5.7.2
+sentry-sdk==2.19.2
+nvidia-cudnn-cu12==9.1.0.70
+fsspec==2024.2.0
+psutil==6.1.0
+pytz==2024.2
+backcall==0.2.0
+nvidia-cusolver-cu12==11.4.5.107
+lxml==5.3.0
+blessed==1.20.0
+pure_eval==0.2.3
+diffusion_policy_3d==0.0.0
+dexart==0.1.0
+gym==0.21.0
+metaworld==0.1.0
+mj-envs==1.0.0
+mjrl==1.0.0
+pytorch3d==0.7.6
+diffusion_policy_3d==0.0.0

adroit_door-dp3-1218_seed0/wandb/wandb-resume.json

@@ -0,0 +1 @@
+{"run_id": "lubq05rh"}

adroit_pen-dp3-1218_seed0/.hydra/overrides.yaml

@@ -0,0 +1,7 @@
+- task=adroit_pen
+- training.debug=False
+- training.seed=0
+- training.device=cuda:0
+- exp_name=adroit_pen-dp3-1218
+- logging.mode=online
+- checkpoint.save_ckpt=True

adroit_pen-dp3-1218_seed0/wandb/run-20241219_022311-wnx82u0h/files/wandb-metadata.json

@@ -0,0 +1,95 @@
+{
+  "os":  "Linux-5.15.0-117-generic-x86_64-with-glibc2.17",
+  "python":  "CPython 3.8.20",
+  "startedAt":  "2024-12-18T18:23:11.064551Z",
+  "args":  [
+    "--config-name=dp3.yaml",
+    "task=adroit_pen",
+    "hydra.run.dir=data/outputs/adroit_pen-dp3-1218_seed0",
+    "training.debug=False",
+    "training.seed=0",
+    "training.device=cuda:0",
+    "exp_name=adroit_pen-dp3-1218",
+    "logging.mode=online",
+    "checkpoint.save_ckpt=True"
+  ],
+  "program":  "train.py",
+  "codePath":  "3D-Diffusion-Policy/train.py",
+  "git":  {
+    "remote":  "git@github.com:YanjieZe/3D-Diffusion-Policy.git",
+    "commit":  "3153fb32c17308fc828015156480b77f6b2e666a"
+  },
+  "email":  "chenhaojun001@gmail.com",
+  "root":  "/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_pen-dp3-1218_seed0",
+  "host":  "Server-3090",
+  "executable":  "/home/chenhaojun/miniconda3/envs/dp3/bin/python",
+  "codePathLocal":  "train.py",
+  "cpu_count":  128,
+  "cpu_count_logical":  256,
+  "gpu":  "NVIDIA GeForce RTX 3090",
+  "gpu_count":  8,
+  "disk":  {
+    "/":  {
+      "total":  "943412031488",
+      "used":  "844653494272"
+    }
+  },
+  "memory":  {
+    "total":  "1081980669952"
+  },
+  "cpu":  {
+    "count":  128,
+    "countLogical":  256
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    }
+  ],
+  "cudaVersion":  "12.5"
+}

adroit_pen-dp3-1218_seed0/wandb/run-20241219_022311-wnx82u0h/files/wandb-summary.json

@@ -0,0 +1 @@
+{"train_action_mse_error":1.8170487237512134e-05,"SR_test_L5":0.465,"_runtime":4569.782587366,"test_mean_score":0.45,"_step":23999,"sim_video_eval":{"size":9413,"path":"media/videos/sim_video_eval_22406_84ba9495019a4765c1ed.mp4","width":84,"height":84,"_type":"video-file","sha256":"84ba9495019a4765c1eda70f83b80b3f7c506340bd48de7d8eb6ecba71f25138"},"train_loss":1.924323089497193e-05,"SR_test_L3":0.48333333333333334,"mean_success_rates":0.45,"lr":0,"epoch":2999,"global_step":23999,"mean_n_goal_achieved":11.85,"_timestamp":1.7345507608399105e+09,"bc_loss":1.4404510693566408e-05,"_wandb":{"runtime":4569}}

adroit_pen-dp3-1218_seed0/wandb/run-20241219_022311-wnx82u0h/logs/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-19T02:23:11.067012008+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-19T02:23:11.067039698+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/adroit_pen-dp3-1218_seed0/wandb/run-20241219_022311-wnx82u0h/logs/debug-core.log"}
+{"time":"2024-12-19T02:23:11.287533236+08:00","level":"INFO","msg":"created new stream","id":"wnx82u0h"}
+{"time":"2024-12-19T02:23:11.287579885+08:00","level":"INFO","msg":"stream: started","id":"wnx82u0h"}
+{"time":"2024-12-19T02:23:11.287650415+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T02:23:11.287714795+08:00","level":"INFO","msg":"handler: started","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T02:23:11.287915284+08:00","level":"INFO","msg":"sender: started","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T02:23:13.851169857+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-19T03:39:20.847103676+08:00","level":"INFO","msg":"stream: closing","id":"wnx82u0h"}
+{"time":"2024-12-19T03:39:20.847167426+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-19T03:39:20.848587386+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-19T03:39:23.680955487+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-19T03:39:26.446790779+08:00","level":"INFO","msg":"handler: closed","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T03:39:26.446866839+08:00","level":"INFO","msg":"sender: closed","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T03:39:26.446853389+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"wnx82u0h"}
+{"time":"2024-12-19T03:39:26.447021069+08:00","level":"INFO","msg":"stream: closed","id":"wnx82u0h"}

adroit_pen-dp3-1218_seed0/wandb/wandb-resume.json

@@ -0,0 +1 @@
+{"run_id": "wnx82u0h"}

dexart_bucket-dp3-1219_seed0/.hydra/config.yaml

@@ -0,0 +1,163 @@
+task:
+  name: dexart_bucket
+  task_name: bucket
+  shape_meta:
+    obs:
+      point_cloud:
+        shape:
+        - 1024
+        - 3
+        type: point_cloud
+      imagin_robot:
+        shape:
+        - 96
+        - 7
+        type: point_cloud
+      image:
+        shape:
+        - 3
+        - 84
+        - 84
+        type: rgb
+      agent_pos:
+        shape:
+        - 33
+        type: low_dim
+    action:
+      shape:
+      - 22
+  env_runner:
+    _target_: diffusion_policy_3d.env_runner.dexart_runner.DexArtRunner
+    n_train: 20
+    max_steps: 50
+    n_obs_steps: ${n_obs_steps}
+    n_action_steps: ${n_action_steps}
+    fps: 10
+    task_name: bucket
+  dataset:
+    _target_: diffusion_policy_3d.dataset.dexart_dataset.DexArtDataset
+    zarr_path: /data/haojun/datasets/3d-dp/dexart_bucket_expert.zarr
+    horizon: ${horizon}
+    pad_before: ${eval:'${n_obs_steps}-1'}
+    pad_after: ${eval:'${n_action_steps}-1'}
+    seed: 42
+    val_ratio: 0.02
+    max_train_episodes: 90
+name: train_dp3
+task_name: ${task.name}
+shape_meta: ${task.shape_meta}
+exp_name: dexart_bucket-dp3-1219
+horizon: 4
+n_obs_steps: 2
+n_action_steps: 3
+n_latency_steps: 0
+dataset_obs_steps: ${n_obs_steps}
+keypoint_visible_rate: 1.0
+obs_as_global_cond: true
+policy:
+  _target_: diffusion_policy_3d.policy.dp3.DP3
+  use_point_crop: true
+  condition_type: film
+  use_down_condition: true
+  use_mid_condition: true
+  use_up_condition: true
+  diffusion_step_embed_dim: 128
+  down_dims:
+  - 512
+  - 1024
+  - 2048
+  crop_shape:
+  - 80
+  - 80
+  encoder_output_dim: 64
+  horizon: ${horizon}
+  kernel_size: 5
+  n_action_steps: ${n_action_steps}
+  n_groups: 8
+  n_obs_steps: ${n_obs_steps}
+  noise_scheduler:
+    _target_: diffusers.schedulers.scheduling_ddim.DDIMScheduler
+    num_train_timesteps: 100
+    beta_start: 0.0001
+    beta_end: 0.02
+    beta_schedule: squaredcos_cap_v2
+    clip_sample: true
+    set_alpha_to_one: true
+    steps_offset: 0
+    prediction_type: sample
+  num_inference_steps: 10
+  obs_as_global_cond: true
+  shape_meta: ${shape_meta}
+  use_pc_color: false
+  pointnet_type: pointnet
+  pointcloud_encoder_cfg:
+    in_channels: 3
+    out_channels: ${policy.encoder_output_dim}
+    use_layernorm: true
+    final_norm: layernorm
+    normal_channel: false
+ema:
+  _target_: diffusion_policy_3d.model.diffusion.ema_model.EMAModel
+  update_after_step: 0
+  inv_gamma: 1.0
+  power: 0.75
+  min_value: 0.0
+  max_value: 0.9999
+dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: true
+  pin_memory: true
+  persistent_workers: false
+val_dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: false
+  pin_memory: true
+  persistent_workers: false
+optimizer:
+  _target_: torch.optim.AdamW
+  lr: 0.0001
+  betas:
+  - 0.95
+  - 0.999
+  eps: 1.0e-08
+  weight_decay: 1.0e-06
+training:
+  device: cuda:0
+  seed: 0
+  debug: false
+  resume: true
+  lr_scheduler: cosine
+  lr_warmup_steps: 500
+  num_epochs: 3000
+  gradient_accumulate_every: 1
+  use_ema: true
+  rollout_every: 200
+  checkpoint_every: 200
+  val_every: 1
+  sample_every: 5
+  max_train_steps: null
+  max_val_steps: null
+  tqdm_interval_sec: 1.0
+logging:
+  group: ${exp_name}
+  id: null
+  mode: online
+  name: ${training.seed}
+  project: dp3
+  resume: true
+  tags:
+  - dp3
+checkpoint:
+  save_ckpt: true
+  topk:
+    monitor_key: test_mean_score
+    mode: max
+    k: 1
+    format_str: epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt
+  save_last_ckpt: true
+  save_last_snapshot: false
+multi_run:
+  run_dir: data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}
+  wandb_name_base: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}

dexart_bucket-dp3-1219_seed0/.hydra/hydra.yaml

@@ -0,0 +1,163 @@
+hydra:
+  run:
+    dir: data/outputs/dexart_bucket-dp3-1219_seed0
+  sweep:
+    dir: data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}
+    subdir: ${hydra.job.num}
+  launcher:
+    _target_: hydra._internal.core_plugins.basic_launcher.BasicLauncher
+  sweeper:
+    _target_: hydra._internal.core_plugins.basic_sweeper.BasicSweeper
+    max_batch_size: null
+    params: null
+  help:
+    app_name: ${hydra.job.name}
+    header: '${hydra.help.app_name} is powered by Hydra.
+
+      '
+    footer: 'Powered by Hydra (https://hydra.cc)
+
+      Use --hydra-help to view Hydra specific help
+
+      '
+    template: '${hydra.help.header}
+
+      == Configuration groups ==
+
+      Compose your configuration from those groups (group=option)
+
+
+      $APP_CONFIG_GROUPS
+
+
+      == Config ==
+
+      Override anything in the config (foo.bar=value)
+
+
+      $CONFIG
+
+
+      ${hydra.help.footer}
+
+      '
+  hydra_help:
+    template: 'Hydra (${hydra.runtime.version})
+
+      See https://hydra.cc for more info.
+
+
+      == Flags ==
+
+      $FLAGS_HELP
+
+
+      == Configuration groups ==
+
+      Compose your configuration from those groups (For example, append hydra/job_logging=disabled
+      to command line)
+
+
+      $HYDRA_CONFIG_GROUPS
+
+
+      Use ''--cfg hydra'' to Show the Hydra config.
+
+      '
+    hydra_help: ???
+  hydra_logging:
+    version: 1
+    formatters:
+      simple:
+        format: '[%(asctime)s][HYDRA] %(message)s'
+    handlers:
+      console:
+        class: logging.StreamHandler
+        formatter: simple
+        stream: ext://sys.stdout
+    root:
+      level: INFO
+      handlers:
+      - console
+    loggers:
+      logging_example:
+        level: DEBUG
+    disable_existing_loggers: false
+  job_logging:
+    version: 1
+    formatters:
+      simple:
+        format: '[%(asctime)s][%(name)s][%(levelname)s] - %(message)s'
+    handlers:
+      console:
+        class: logging.StreamHandler
+        formatter: simple
+        stream: ext://sys.stdout
+      file:
+        class: logging.FileHandler
+        formatter: simple
+        filename: ${hydra.runtime.output_dir}/${hydra.job.name}.log
+    root:
+      level: INFO
+      handlers:
+      - console
+      - file
+    disable_existing_loggers: false
+  env: {}
+  mode: RUN
+  searchpath: []
+  callbacks: {}
+  output_subdir: .hydra
+  overrides:
+    hydra:
+    - hydra.run.dir=data/outputs/dexart_bucket-dp3-1219_seed0
+    - hydra.mode=RUN
+    task:
+    - task=dexart_bucket
+    - training.debug=False
+    - training.seed=0
+    - training.device=cuda:0
+    - exp_name=dexart_bucket-dp3-1219
+    - logging.mode=online
+    - checkpoint.save_ckpt=True
+  job:
+    name: train
+    chdir: null
+    override_dirname: checkpoint.save_ckpt=True,exp_name=dexart_bucket-dp3-1219,logging.mode=online,task=dexart_bucket,training.debug=False,training.device=cuda:0,training.seed=0
+    id: ???
+    num: ???
+    config_name: dp3.yaml
+    env_set: {}
+    env_copy: []
+    config:
+      override_dirname:
+        kv_sep: '='
+        item_sep: ','
+        exclude_keys: []
+  runtime:
+    version: 1.2.0
+    version_base: '1.2'
+    cwd: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy
+    config_sources:
+    - path: hydra.conf
+      schema: pkg
+      provider: hydra
+    - path: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/config
+      schema: file
+      provider: main
+    - path: ''
+      schema: structured
+      provider: schema
+    output_dir: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/dexart_bucket-dp3-1219_seed0
+    choices:
+      task: dexart_bucket
+      hydra/env: default
+      hydra/callbacks: null
+      hydra/job_logging: default
+      hydra/hydra_logging: default
+      hydra/hydra_help: default
+      hydra/help: default
+      hydra/sweeper: basic
+      hydra/launcher: basic
+      hydra/output: default
+  verbose: false

metaworld_assembly-dp3-1218_seed0/wandb/debug.log

@@ -0,0 +1,24 @@
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_setup.py:_flush():68] Configure stats pid to 2428026
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_assembly-dp3-1218_seed0/wandb/run-20241218_142852-63943iru/logs/debug.log
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_assembly-dp3-1218_seed0/wandb/run-20241218_142852-63943iru/logs/debug-internal.log
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:init():644] calling init triggers
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'assembly', 'task_name': 'train_dp3', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner', 'eval_episodes': 20, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'n_envs': None, 'n_train': None, 'n_test': None, 'task_name': 'assembly', 'device': 'cuda:0', 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/metaworld_assembly_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'assembly', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'exp_name': 'metaworld_assembly-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'metaworld_assembly-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.18/14.28.49_train_dp3_assembly', 'wandb_name_base': '2024.12.18-14.28.49_train_dp3_assembly'}}
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:init():680] starting backend
+2024-12-18 14:28:52,369 INFO    MainThread:2428026 [wandb_init.py:init():684] sending inform_init request
+2024-12-18 14:28:52,395 INFO    MainThread:2428026 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-18 14:28:52,395 INFO    MainThread:2428026 [wandb_init.py:init():697] backend started and connected
+2024-12-18 14:28:52,400 INFO    MainThread:2428026 [wandb_init.py:init():790] updated telemetry
+2024-12-18 14:28:52,577 INFO    MainThread:2428026 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-18 14:28:55,105 INFO    MainThread:2428026 [wandb_init.py:init():874] starting run threads in backend
+2024-12-18 14:28:55,225 INFO    MainThread:2428026 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-18 14:28:55,225 INFO    MainThread:2428026 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-18 14:28:55,226 INFO    MainThread:2428026 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-18 14:28:55,226 INFO    MainThread:2428026 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-18 14:28:55,228 INFO    MainThread:2428026 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-18 14:28:55,228 INFO    MainThread:2428026 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_assembly-dp3-1218_seed0'}
+2024-12-18 16:42:47,860 WARNING MsgRouterThr:2428026 [router.py:message_loop():75] message_loop has been closed

metaworld_assembly-dp3-1218_seed0/wandb/run-20241218_142852-63943iru/logs/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-18T14:28:52.397338454+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-18T14:28:52.397351034+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_assembly-dp3-1218_seed0/wandb/run-20241218_142852-63943iru/logs/debug-core.log"}
+{"time":"2024-12-18T14:28:52.609650827+08:00","level":"INFO","msg":"created new stream","id":"63943iru"}
+{"time":"2024-12-18T14:28:52.609680037+08:00","level":"INFO","msg":"stream: started","id":"63943iru"}
+{"time":"2024-12-18T14:28:52.609755478+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"63943iru"}
+{"time":"2024-12-18T14:28:52.609795528+08:00","level":"INFO","msg":"handler: started","stream_id":"63943iru"}
+{"time":"2024-12-18T14:28:52.611313401+08:00","level":"INFO","msg":"sender: started","stream_id":"63943iru"}
+{"time":"2024-12-18T14:28:55.113722346+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-18T16:42:47.860321056+08:00","level":"INFO","msg":"stream: closing","id":"63943iru"}
+{"time":"2024-12-18T16:42:47.860366545+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-18T16:42:47.861825165+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-18T16:42:51.203826373+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-18T16:42:54.393251212+08:00","level":"INFO","msg":"handler: closed","stream_id":"63943iru"}
+{"time":"2024-12-18T16:42:54.393317331+08:00","level":"INFO","msg":"sender: closed","stream_id":"63943iru"}
+{"time":"2024-12-18T16:42:54.393312931+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"63943iru"}
+{"time":"2024-12-18T16:42:54.393483768+08:00","level":"INFO","msg":"stream: closed","id":"63943iru"}

metaworld_basketball-dp3-1218_seed0/.hydra/config.yaml

@@ -0,0 +1,156 @@
+task:
+  name: basketball
+  task_name: ${name}
+  shape_meta:
+    obs:
+      point_cloud:
+        shape:
+        - 512
+        - 3
+        type: point_cloud
+      agent_pos:
+        shape:
+        - 9
+        type: low_dim
+    action:
+      shape:
+      - 4
+  env_runner:
+    _target_: diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner
+    eval_episodes: 20
+    n_obs_steps: ${n_obs_steps}
+    n_action_steps: ${n_action_steps}
+    fps: 10
+    n_envs: null
+    n_train: null
+    n_test: null
+    task_name: ${task_name}
+    device: ${training.device}
+    use_point_crop: ${policy.use_point_crop}
+  dataset:
+    _target_: diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset
+    zarr_path: /data/haojun/datasets/3d-dp/metaworld_basketball_expert.zarr
+    horizon: ${horizon}
+    pad_before: ${eval:'${n_obs_steps}-1'}
+    pad_after: ${eval:'${n_action_steps}-1'}
+    seed: 42
+    val_ratio: 0.02
+    max_train_episodes: 90
+name: train_dp3
+task_name: ${task.name}
+shape_meta: ${task.shape_meta}
+exp_name: metaworld_basketball-dp3-1218
+horizon: 4
+n_obs_steps: 2
+n_action_steps: 3
+n_latency_steps: 0
+dataset_obs_steps: ${n_obs_steps}
+keypoint_visible_rate: 1.0
+obs_as_global_cond: true
+policy:
+  _target_: diffusion_policy_3d.policy.dp3.DP3
+  use_point_crop: true
+  condition_type: film
+  use_down_condition: true
+  use_mid_condition: true
+  use_up_condition: true
+  diffusion_step_embed_dim: 128
+  down_dims:
+  - 512
+  - 1024
+  - 2048
+  crop_shape:
+  - 80
+  - 80
+  encoder_output_dim: 64
+  horizon: ${horizon}
+  kernel_size: 5
+  n_action_steps: ${n_action_steps}
+  n_groups: 8
+  n_obs_steps: ${n_obs_steps}
+  noise_scheduler:
+    _target_: diffusers.schedulers.scheduling_ddim.DDIMScheduler
+    num_train_timesteps: 100
+    beta_start: 0.0001
+    beta_end: 0.02
+    beta_schedule: squaredcos_cap_v2
+    clip_sample: true
+    set_alpha_to_one: true
+    steps_offset: 0
+    prediction_type: sample
+  num_inference_steps: 10
+  obs_as_global_cond: true
+  shape_meta: ${shape_meta}
+  use_pc_color: false
+  pointnet_type: pointnet
+  pointcloud_encoder_cfg:
+    in_channels: 3
+    out_channels: ${policy.encoder_output_dim}
+    use_layernorm: true
+    final_norm: layernorm
+    normal_channel: false
+ema:
+  _target_: diffusion_policy_3d.model.diffusion.ema_model.EMAModel
+  update_after_step: 0
+  inv_gamma: 1.0
+  power: 0.75
+  min_value: 0.0
+  max_value: 0.9999
+dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: true
+  pin_memory: true
+  persistent_workers: false
+val_dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: false
+  pin_memory: true
+  persistent_workers: false
+optimizer:
+  _target_: torch.optim.AdamW
+  lr: 0.0001
+  betas:
+  - 0.95
+  - 0.999
+  eps: 1.0e-08
+  weight_decay: 1.0e-06
+training:
+  device: cuda:0
+  seed: 0
+  debug: false
+  resume: true
+  lr_scheduler: cosine
+  lr_warmup_steps: 500
+  num_epochs: 3000
+  gradient_accumulate_every: 1
+  use_ema: true
+  rollout_every: 200
+  checkpoint_every: 200
+  val_every: 1
+  sample_every: 5
+  max_train_steps: null
+  max_val_steps: null
+  tqdm_interval_sec: 1.0
+logging:
+  group: ${exp_name}
+  id: null
+  mode: online
+  name: ${training.seed}
+  project: dp3
+  resume: true
+  tags:
+  - dp3
+checkpoint:
+  save_ckpt: true
+  topk:
+    monitor_key: test_mean_score
+    mode: max
+    k: 1
+    format_str: epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt
+  save_last_ckpt: true
+  save_last_snapshot: false
+multi_run:
+  run_dir: data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}
+  wandb_name_base: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}

metaworld_basketball-dp3-1218_seed0/.hydra/overrides.yaml

@@ -0,0 +1,7 @@
+- task=metaworld_basketball
+- training.debug=False
+- training.seed=0
+- training.device=cuda:0
+- exp_name=metaworld_basketball-dp3-1218
+- logging.mode=online
+- checkpoint.save_ckpt=True

metaworld_basketball-dp3-1218_seed0/train.log

@@ -0,0 +1 @@
+[2024-12-19 02:12:45,544][diffusion_policy_3d.model.diffusion.conditional_unet1d][INFO] - number of parameters: 2.550744e+08

metaworld_basketball-dp3-1218_seed0/wandb/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-19T02:12:53.370257505+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-19T02:12:53.370283045+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug-core.log"}
+{"time":"2024-12-19T02:12:53.590994224+08:00","level":"INFO","msg":"created new stream","id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591036474+08:00","level":"INFO","msg":"stream: started","id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591118355+08:00","level":"INFO","msg":"sender: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591118455+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591162465+08:00","level":"INFO","msg":"handler: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:56.150864635+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-19T04:23:59.855001468+08:00","level":"INFO","msg":"stream: closing","id":"0yt2mdpc"}
+{"time":"2024-12-19T04:23:59.855049448+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-19T04:23:59.856264931+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-19T04:24:03.004930054+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-19T04:24:03.979616847+08:00","level":"INFO","msg":"handler: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979669597+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979711507+08:00","level":"INFO","msg":"sender: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979837676+08:00","level":"INFO","msg":"stream: closed","id":"0yt2mdpc"}

metaworld_basketball-dp3-1218_seed0/wandb/debug.log

@@ -0,0 +1,24 @@
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Configure stats pid to 4133489
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug.log
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug-internal.log
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():644] calling init triggers
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'basketball', 'task_name': 'train_dp3', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner', 'eval_episodes': 20, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'n_envs': None, 'n_train': None, 'n_test': None, 'task_name': 'basketball', 'device': 'cuda:0', 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/metaworld_basketball_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'basketball', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'exp_name': 'metaworld_basketball-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'metaworld_basketball-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.19/02.12.51_train_dp3_basketball', 'wandb_name_base': '2024.12.19-02.12.51_train_dp3_basketball'}}
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():680] starting backend
+2024-12-19 02:12:53,340 INFO    MainThread:4133489 [wandb_init.py:init():684] sending inform_init request
+2024-12-19 02:12:53,366 INFO    MainThread:4133489 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-19 02:12:53,367 INFO    MainThread:4133489 [wandb_init.py:init():697] backend started and connected
+2024-12-19 02:12:53,372 INFO    MainThread:4133489 [wandb_init.py:init():790] updated telemetry
+2024-12-19 02:12:53,552 INFO    MainThread:4133489 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-19 02:12:56,141 INFO    MainThread:4133489 [wandb_init.py:init():874] starting run threads in backend
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-19 02:12:56,269 INFO    MainThread:4133489 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-19 02:12:56,269 INFO    MainThread:4133489 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0'}
+2024-12-19 04:23:59,855 WARNING MsgRouterThr:4133489 [router.py:message_loop():75] message_loop has been closed

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/files/output.log

@@ -0,0 +1,46 @@
+                                                                                                                             
+test_mean_score: 0.0
+mean_durations: 4.811403276212514
+std_durations: 0.014237352510659471
+test_mean_score: 1.0
+mean_durations: 4.842985672783106
+std_durations: 0.005584776641334111
+test_mean_score: 1.0
+mean_durations: 4.839315736945719
+std_durations: 0.0031868470379062394
+test_mean_score: 1.0
+mean_durations: 4.84056283608079
+std_durations: 0.011284331881369942
+test_mean_score: 1.0
+mean_durations: 4.83910873895511
+std_durations: 0.011735428375009465
+test_mean_score: 1.0
+mean_durations: 4.842095082998275
+std_durations: 0.0058578681648754054
+test_mean_score: 1.0
+mean_durations: 4.840674304310232
+std_durations: 0.007631154396389606
+test_mean_score: 1.0
+mean_durations: 4.919937895052135
+std_durations: 0.22635758178988977
+test_mean_score: 1.0
+mean_durations: 4.846944478619844
+std_durations: 0.016485908985989735
+test_mean_score: 1.0
+mean_durations: 4.843696603551507
+std_durations: 0.006973857450969386
+test_mean_score: 1.0
+mean_durations: 4.835033345874399
+std_durations: 0.007871813911208983
+test_mean_score: 1.0
+mean_durations: 4.832790290098638
+std_durations: 0.007823855037154251
+test_mean_score: 1.0
+mean_durations: 4.833300226274877
+std_durations: 0.0030746198449928142
+test_mean_score: 1.0
+mean_durations: 4.830803212244064
+std_durations: 0.005497308323561335
+test_mean_score: 1.0
+mean_durations: 4.8261617921292785
+std_durations: 0.003801503208100719

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/files/requirements.txt

@@ -0,0 +1,175 @@
+diffusion_policy_3d==0.0.0
+certifi==2024.12.14
+PyYAML==6.0.2
+dash-table==5.0.0
+dm_control==1.0.23
+einops==0.4.1
+ptyprocess==0.7.0
+jsonschema==4.23.0
+prompt_toolkit==3.0.48
+nbformat==5.10.4
+decorator==4.4.2
+importlib_metadata==8.5.0
+numcodecs==0.12.1
+requests==2.32.3
+ipywidgets==8.1.5
+urllib3==2.2.3
+platformdirs==4.3.6
+imageio-ffmpeg==0.5.1
+dash==2.18.2
+Cython==0.29.35
+mujoco-py==2.1.2.14
+mpmath==1.3.0
+typing_extensions==4.12.2
+dm-tree==0.1.8
+imageio==2.35.1
+dill==0.3.5.1
+joblib==1.4.2
+attrs==24.3.0
+torchvision==0.19.1+cu121
+kiwisolver==1.4.7
+numpy==1.23.5
+pillow==10.2.0
+nvidia-cuda-cupti-cu12==12.1.105
+pip==24.2
+llvmlite==0.39.1
+tenacity==9.0.0
+scipy==1.10.1
+charset-normalizer==3.4.0
+matplotlib==3.7.5
+jupyterlab_widgets==3.0.13
+patchelf==0.17.2.0
+wcwidth==0.2.13
+pexpect==4.9.0
+plotly==5.24.1
+ipdb==0.13.13
+referencing==0.35.1
+mujoco==3.2.3
+portalocker==3.0.0
+jedi==0.19.2
+Jinja2==3.1.3
+MarkupSafe==2.1.5
+nvidia-nvtx-cu12==12.1.105
+python-dateutil==2.9.0.post0
+packaging==24.2
+click==8.1.7
+pycparser==2.22
+blinker==1.8.2
+sympy==1.13.1
+Werkzeug==3.0.6
+six==1.17.0
+stack-data==0.6.3
+gpustat==1.1.1
+threadpoolctl==3.5.0
+traitlets==5.14.3
+annotated-types==0.7.0
+nvidia-nvjitlink-cu12==12.1.105
+zarr==2.12.0
+pandas==2.0.3
+wheel==0.44.0
+nvidia-nccl-cu12==2.20.5
+comm==0.2.2
+proglog==0.1.10
+parso==0.8.4
+open3d==0.18.0
+nvidia-cusparse-cu12==12.1.0.106
+antlr4-python3-runtime==4.9.3
+fvcore==0.1.5.post20221221
+natsort==8.4.0
+cloudpickle==3.1.0
+ConfigArgParse==1.7
+fonttools==4.55.3
+etils==1.3.0
+torchaudio==2.4.1+cu121
+pickleshare==0.7.5
+eval_type_backport==0.2.0
+executing==2.1.0
+addict==2.4.0
+protobuf==5.29.1
+setproctitle==1.3.4
+Flask==3.0.3
+filelock==3.13.1
+regex==2024.11.6
+PyOpenGL==3.1.7
+nvidia-curand-cu12==10.3.2.106
+matplotlib-inline==0.1.7
+transforms3d==0.4.2
+pydantic==2.10.3
+cffi==1.17.1
+rpds-py==0.20.1
+diffusers==0.11.1
+docker-pycreds==0.4.0
+pyparsing==3.1.4
+absl-py==2.1.0
+nvidia-cufft-cu12==11.0.2.54
+contourpy==1.1.1
+fastjsonschema==2.21.1
+pkgutil_resolve_name==1.3.10
+dash-html-components==2.0.0
+asciitree==0.3.3
+fasteners==0.19
+glfw==2.8.0
+omegaconf==2.3.0
+asttokens==3.0.0
+moviepy==1.0.3
+GitPython==3.1.43
+opencv-python==4.10.0.84
+zipp==3.20.2
+tabulate==0.9.0
+iopath==0.1.10
+labmaze==1.0.6
+numba==0.56.4
+nvidia-cuda-runtime-cu12==12.1.105
+dm-env==1.6
+cycler==0.12.1
+dash-core-components==2.0.0
+networkx==3.0
+Pygments==2.18.0
+retrying==1.3.4
+jsonschema-specifications==2023.12.1
+setuptools==59.5.0
+av==12.3.0
+itsdangerous==2.2.0
+wandb==0.19.1
+nest-asyncio==1.6.0
+triton==3.0.0
+nvidia-ml-py==12.560.30
+nvidia-cuda-nvrtc-cu12==12.1.105
+tomli==2.2.1
+tqdm==4.67.1
+termcolor==2.4.0
+hydra-core==1.2.0
+smmap==5.0.1
+ipython==8.12.3
+pyquaternion==0.9.9
+idna==3.10
+yacs==0.1.8
+tzdata==2024.2
+torch==2.4.1+cu121
+widgetsnbextension==4.0.13
+scikit-learn==1.3.2
+sapien==2.2.1
+nvidia-cublas-cu12==12.1.3.1
+huggingface-hub==0.21.0
+gitdb==4.0.11
+importlib_resources==6.4.5
+pydantic_core==2.27.1
+jupyter_core==5.7.2
+sentry-sdk==2.19.2
+nvidia-cudnn-cu12==9.1.0.70
+fsspec==2024.2.0
+psutil==6.1.0
+pytz==2024.2
+backcall==0.2.0
+nvidia-cusolver-cu12==11.4.5.107
+lxml==5.3.0
+blessed==1.20.0
+pure_eval==0.2.3
+diffusion_policy_3d==0.0.0
+dexart==0.1.0
+gym==0.21.0
+metaworld==0.1.0
+mj-envs==1.0.0
+mjrl==1.0.0
+pytorch3d==0.7.6
+diffusion_policy_3d==0.0.0

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/files/wandb-metadata.json

@@ -0,0 +1,95 @@
+{
+  "os":  "Linux-5.15.0-117-generic-x86_64-with-glibc2.17",
+  "python":  "CPython 3.8.20",
+  "startedAt":  "2024-12-18T18:12:53.367588Z",
+  "args":  [
+    "--config-name=dp3.yaml",
+    "task=metaworld_basketball",
+    "hydra.run.dir=data/outputs/metaworld_basketball-dp3-1218_seed0",
+    "training.debug=False",
+    "training.seed=0",
+    "training.device=cuda:0",
+    "exp_name=metaworld_basketball-dp3-1218",
+    "logging.mode=online",
+    "checkpoint.save_ckpt=True"
+  ],
+  "program":  "train.py",
+  "codePath":  "3D-Diffusion-Policy/train.py",
+  "git":  {
+    "remote":  "git@github.com:YanjieZe/3D-Diffusion-Policy.git",
+    "commit":  "3153fb32c17308fc828015156480b77f6b2e666a"
+  },
+  "email":  "chenhaojun001@gmail.com",
+  "root":  "/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0",
+  "host":  "Server-3090",
+  "executable":  "/home/chenhaojun/miniconda3/envs/dp3/bin/python",
+  "codePathLocal":  "train.py",
+  "cpu_count":  128,
+  "cpu_count_logical":  256,
+  "gpu":  "NVIDIA GeForce RTX 3090",
+  "gpu_count":  8,
+  "disk":  {
+    "/":  {
+      "total":  "943412031488",
+      "used":  "821133565952"
+    }
+  },
+  "memory":  {
+    "total":  "1081980669952"
+  },
+  "cpu":  {
+    "count":  128,
+    "countLogical":  256
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    }
+  ],
+  "cudaVersion":  "12.5"
+}

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/files/wandb-summary.json

@@ -0,0 +1 @@
+{"train_action_mse_error":9.559800673741847e-05,"SR_test_L5":1,"_timestamp":1.7345534398475282e+09,"train_loss":1.2388219677935317e-06,"_runtime":7866.487449238,"std_durations":0.003801503208100719,"SR_test_L3":1,"global_step":44999,"lr":0,"bc_loss":9.440128678761539e-07,"mean_success_rates":1,"mean_durations":4.8261617921292785,"_step":44999,"_wandb":{"runtime":7866},"mean_traj_rewards":21534.710823107216,"epoch":2999,"test_mean_score":1}

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-19T02:12:53.370257505+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-19T02:12:53.370283045+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug-core.log"}
+{"time":"2024-12-19T02:12:53.590994224+08:00","level":"INFO","msg":"created new stream","id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591036474+08:00","level":"INFO","msg":"stream: started","id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591118355+08:00","level":"INFO","msg":"sender: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591118455+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:53.591162465+08:00","level":"INFO","msg":"handler: started","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T02:12:56.150864635+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-19T04:23:59.855001468+08:00","level":"INFO","msg":"stream: closing","id":"0yt2mdpc"}
+{"time":"2024-12-19T04:23:59.855049448+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-19T04:23:59.856264931+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-19T04:24:03.004930054+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-19T04:24:03.979616847+08:00","level":"INFO","msg":"handler: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979669597+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979711507+08:00","level":"INFO","msg":"sender: closed","stream_id":"0yt2mdpc"}
+{"time":"2024-12-19T04:24:03.979837676+08:00","level":"INFO","msg":"stream: closed","id":"0yt2mdpc"}

metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug.log

@@ -0,0 +1,24 @@
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Configure stats pid to 4133489
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug.log
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0/wandb/run-20241219_021253-0yt2mdpc/logs/debug-internal.log
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():644] calling init triggers
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'basketball', 'task_name': 'train_dp3', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner', 'eval_episodes': 20, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'n_envs': None, 'n_train': None, 'n_test': None, 'task_name': 'basketball', 'device': 'cuda:0', 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/metaworld_basketball_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'basketball', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'exp_name': 'metaworld_basketball-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'metaworld_basketball-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.19/02.12.51_train_dp3_basketball', 'wandb_name_base': '2024.12.19-02.12.51_train_dp3_basketball'}}
+2024-12-19 02:12:53,339 INFO    MainThread:4133489 [wandb_init.py:init():680] starting backend
+2024-12-19 02:12:53,340 INFO    MainThread:4133489 [wandb_init.py:init():684] sending inform_init request
+2024-12-19 02:12:53,366 INFO    MainThread:4133489 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-19 02:12:53,367 INFO    MainThread:4133489 [wandb_init.py:init():697] backend started and connected
+2024-12-19 02:12:53,372 INFO    MainThread:4133489 [wandb_init.py:init():790] updated telemetry
+2024-12-19 02:12:53,552 INFO    MainThread:4133489 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-19 02:12:56,141 INFO    MainThread:4133489 [wandb_init.py:init():874] starting run threads in backend
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-19 02:12:56,267 INFO    MainThread:4133489 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-19 02:12:56,269 INFO    MainThread:4133489 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-19 02:12:56,269 INFO    MainThread:4133489 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_basketball-dp3-1218_seed0'}
+2024-12-19 04:23:59,855 WARNING MsgRouterThr:4133489 [router.py:message_loop():75] message_loop has been closed

metaworld_basketball-dp3-1218_seed0/wandb/wandb-resume.json

@@ -0,0 +1 @@
+{"run_id": "0yt2mdpc"}

metaworld_button-press-dp3-1218_seed0/.hydra/config.yaml

@@ -0,0 +1,156 @@
+task:
+  name: button-press
+  task_name: ${name}
+  shape_meta:
+    obs:
+      point_cloud:
+        shape:
+        - 512
+        - 3
+        type: point_cloud
+      agent_pos:
+        shape:
+        - 9
+        type: low_dim
+    action:
+      shape:
+      - 4
+  env_runner:
+    _target_: diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner
+    eval_episodes: 20
+    n_obs_steps: ${n_obs_steps}
+    n_action_steps: ${n_action_steps}
+    fps: 10
+    n_envs: null
+    n_train: null
+    n_test: null
+    task_name: ${task_name}
+    device: ${training.device}
+    use_point_crop: ${policy.use_point_crop}
+  dataset:
+    _target_: diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset
+    zarr_path: /data/haojun/datasets/3d-dp/metaworld_button-press_expert.zarr
+    horizon: ${horizon}
+    pad_before: ${eval:'${n_obs_steps}-1'}
+    pad_after: ${eval:'${n_action_steps}-1'}
+    seed: 42
+    val_ratio: 0.02
+    max_train_episodes: 90
+name: train_dp3
+task_name: ${task.name}
+shape_meta: ${task.shape_meta}
+exp_name: metaworld_button-press-dp3-1218
+horizon: 4
+n_obs_steps: 2
+n_action_steps: 3
+n_latency_steps: 0
+dataset_obs_steps: ${n_obs_steps}
+keypoint_visible_rate: 1.0
+obs_as_global_cond: true
+policy:
+  _target_: diffusion_policy_3d.policy.dp3.DP3
+  use_point_crop: true
+  condition_type: film
+  use_down_condition: true
+  use_mid_condition: true
+  use_up_condition: true
+  diffusion_step_embed_dim: 128
+  down_dims:
+  - 512
+  - 1024
+  - 2048
+  crop_shape:
+  - 80
+  - 80
+  encoder_output_dim: 64
+  horizon: ${horizon}
+  kernel_size: 5
+  n_action_steps: ${n_action_steps}
+  n_groups: 8
+  n_obs_steps: ${n_obs_steps}
+  noise_scheduler:
+    _target_: diffusers.schedulers.scheduling_ddim.DDIMScheduler
+    num_train_timesteps: 100
+    beta_start: 0.0001
+    beta_end: 0.02
+    beta_schedule: squaredcos_cap_v2
+    clip_sample: true
+    set_alpha_to_one: true
+    steps_offset: 0
+    prediction_type: sample
+  num_inference_steps: 10
+  obs_as_global_cond: true
+  shape_meta: ${shape_meta}
+  use_pc_color: false
+  pointnet_type: pointnet
+  pointcloud_encoder_cfg:
+    in_channels: 3
+    out_channels: ${policy.encoder_output_dim}
+    use_layernorm: true
+    final_norm: layernorm
+    normal_channel: false
+ema:
+  _target_: diffusion_policy_3d.model.diffusion.ema_model.EMAModel
+  update_after_step: 0
+  inv_gamma: 1.0
+  power: 0.75
+  min_value: 0.0
+  max_value: 0.9999
+dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: true
+  pin_memory: true
+  persistent_workers: false
+val_dataloader:
+  batch_size: 128
+  num_workers: 8
+  shuffle: false
+  pin_memory: true
+  persistent_workers: false
+optimizer:
+  _target_: torch.optim.AdamW
+  lr: 0.0001
+  betas:
+  - 0.95
+  - 0.999
+  eps: 1.0e-08
+  weight_decay: 1.0e-06
+training:
+  device: cuda:0
+  seed: 0
+  debug: false
+  resume: true
+  lr_scheduler: cosine
+  lr_warmup_steps: 500
+  num_epochs: 3000
+  gradient_accumulate_every: 1
+  use_ema: true
+  rollout_every: 200
+  checkpoint_every: 200
+  val_every: 1
+  sample_every: 5
+  max_train_steps: null
+  max_val_steps: null
+  tqdm_interval_sec: 1.0
+logging:
+  group: ${exp_name}
+  id: null
+  mode: online
+  name: ${training.seed}
+  project: dp3
+  resume: true
+  tags:
+  - dp3
+checkpoint:
+  save_ckpt: true
+  topk:
+    monitor_key: test_mean_score
+    mode: max
+    k: 1
+    format_str: epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt
+  save_last_ckpt: true
+  save_last_snapshot: false
+multi_run:
+  run_dir: data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}
+  wandb_name_base: ${now:%Y.%m.%d-%H.%M.%S}_${name}_${task_name}

metaworld_button-press-dp3-1218_seed0/.hydra/hydra.yaml

@@ -0,0 +1,163 @@
+hydra:
+  run:
+    dir: data/outputs/metaworld_button-press-dp3-1218_seed0
+  sweep:
+    dir: data/outputs/${now:%Y.%m.%d}/${now:%H.%M.%S}_${name}_${task_name}
+    subdir: ${hydra.job.num}
+  launcher:
+    _target_: hydra._internal.core_plugins.basic_launcher.BasicLauncher
+  sweeper:
+    _target_: hydra._internal.core_plugins.basic_sweeper.BasicSweeper
+    max_batch_size: null
+    params: null
+  help:
+    app_name: ${hydra.job.name}
+    header: '${hydra.help.app_name} is powered by Hydra.
+
+      '
+    footer: 'Powered by Hydra (https://hydra.cc)
+
+      Use --hydra-help to view Hydra specific help
+
+      '
+    template: '${hydra.help.header}
+
+      == Configuration groups ==
+
+      Compose your configuration from those groups (group=option)
+
+
+      $APP_CONFIG_GROUPS
+
+
+      == Config ==
+
+      Override anything in the config (foo.bar=value)
+
+
+      $CONFIG
+
+
+      ${hydra.help.footer}
+
+      '
+  hydra_help:
+    template: 'Hydra (${hydra.runtime.version})
+
+      See https://hydra.cc for more info.
+
+
+      == Flags ==
+
+      $FLAGS_HELP
+
+
+      == Configuration groups ==
+
+      Compose your configuration from those groups (For example, append hydra/job_logging=disabled
+      to command line)
+
+
+      $HYDRA_CONFIG_GROUPS
+
+
+      Use ''--cfg hydra'' to Show the Hydra config.
+
+      '
+    hydra_help: ???
+  hydra_logging:
+    version: 1
+    formatters:
+      simple:
+        format: '[%(asctime)s][HYDRA] %(message)s'
+    handlers:
+      console:
+        class: logging.StreamHandler
+        formatter: simple
+        stream: ext://sys.stdout
+    root:
+      level: INFO
+      handlers:
+      - console
+    loggers:
+      logging_example:
+        level: DEBUG
+    disable_existing_loggers: false
+  job_logging:
+    version: 1
+    formatters:
+      simple:
+        format: '[%(asctime)s][%(name)s][%(levelname)s] - %(message)s'
+    handlers:
+      console:
+        class: logging.StreamHandler
+        formatter: simple
+        stream: ext://sys.stdout
+      file:
+        class: logging.FileHandler
+        formatter: simple
+        filename: ${hydra.runtime.output_dir}/${hydra.job.name}.log
+    root:
+      level: INFO
+      handlers:
+      - console
+      - file
+    disable_existing_loggers: false
+  env: {}
+  mode: RUN
+  searchpath: []
+  callbacks: {}
+  output_subdir: .hydra
+  overrides:
+    hydra:
+    - hydra.run.dir=data/outputs/metaworld_button-press-dp3-1218_seed0
+    - hydra.mode=RUN
+    task:
+    - task=metaworld_button-press
+    - training.debug=False
+    - training.seed=0
+    - training.device=cuda:0
+    - exp_name=metaworld_button-press-dp3-1218
+    - logging.mode=online
+    - checkpoint.save_ckpt=True
+  job:
+    name: train
+    chdir: null
+    override_dirname: checkpoint.save_ckpt=True,exp_name=metaworld_button-press-dp3-1218,logging.mode=online,task=metaworld_button-press,training.debug=False,training.device=cuda:0,training.seed=0
+    id: ???
+    num: ???
+    config_name: dp3.yaml
+    env_set: {}
+    env_copy: []
+    config:
+      override_dirname:
+        kv_sep: '='
+        item_sep: ','
+        exclude_keys: []
+  runtime:
+    version: 1.2.0
+    version_base: '1.2'
+    cwd: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy
+    config_sources:
+    - path: hydra.conf
+      schema: pkg
+      provider: hydra
+    - path: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/config
+      schema: file
+      provider: main
+    - path: ''
+      schema: structured
+      provider: schema
+    output_dir: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0
+    choices:
+      task: metaworld_button-press
+      hydra/env: default
+      hydra/callbacks: null
+      hydra/job_logging: default
+      hydra/hydra_logging: default
+      hydra/hydra_help: default
+      hydra/help: default
+      hydra/sweeper: basic
+      hydra/launcher: basic
+      hydra/output: default
+  verbose: false

metaworld_button-press-dp3-1218_seed0/.hydra/overrides.yaml

@@ -0,0 +1,7 @@
+- task=metaworld_button-press
+- training.debug=False
+- training.seed=0
+- training.device=cuda:0
+- exp_name=metaworld_button-press-dp3-1218
+- logging.mode=online
+- checkpoint.save_ckpt=True

metaworld_button-press-dp3-1218_seed0/checkpoints/info.txt

@@ -0,0 +1 @@
+epoch=0200-test_mean_score=1.000.ckpt

metaworld_button-press-dp3-1218_seed0/train.log

@@ -0,0 +1 @@
+[2024-12-18 20:53:11,897][diffusion_policy_3d.model.diffusion.conditional_unet1d][INFO] - number of parameters: 2.550744e+08

metaworld_button-press-dp3-1218_seed0/wandb/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-18T20:53:19.200306453+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-18T20:53:19.200319883+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug-core.log"}
+{"time":"2024-12-18T20:53:19.41309207+08:00","level":"INFO","msg":"created new stream","id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413148729+08:00","level":"INFO","msg":"stream: started","id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413198479+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413237219+08:00","level":"INFO","msg":"sender: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413334339+08:00","level":"INFO","msg":"handler: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:21.942657185+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-19T00:14:16.150854271+08:00","level":"INFO","msg":"stream: closing","id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:16.150900131+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-19T00:14:16.152117904+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-19T00:14:19.106555074+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-19T00:14:20.252272164+08:00","level":"INFO","msg":"handler: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252345544+08:00","level":"INFO","msg":"sender: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252336774+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252507633+08:00","level":"INFO","msg":"stream: closed","id":"rdsa8ydn"}

metaworld_button-press-dp3-1218_seed0/wandb/debug.log

@@ -0,0 +1,24 @@
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Configure stats pid to 2963634
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug.log
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug-internal.log
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():644] calling init triggers
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'button-press', 'task_name': 'train_dp3', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner', 'eval_episodes': 20, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'n_envs': None, 'n_train': None, 'n_test': None, 'task_name': 'button-press', 'device': 'cuda:0', 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/metaworld_button-press_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'button-press', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'exp_name': 'metaworld_button-press-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'metaworld_button-press-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.18/20.53.17_train_dp3_button-press', 'wandb_name_base': '2024.12.18-20.53.17_train_dp3_button-press'}}
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():680] starting backend
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():684] sending inform_init request
+2024-12-18 20:53:19,198 INFO    MainThread:2963634 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-18 20:53:19,198 INFO    MainThread:2963634 [wandb_init.py:init():697] backend started and connected
+2024-12-18 20:53:19,203 INFO    MainThread:2963634 [wandb_init.py:init():790] updated telemetry
+2024-12-18 20:53:19,347 INFO    MainThread:2963634 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-18 20:53:21,937 INFO    MainThread:2963634 [wandb_init.py:init():874] starting run threads in backend
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-18 20:53:22,054 INFO    MainThread:2963634 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-18 20:53:22,055 INFO    MainThread:2963634 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0'}
+2024-12-19 00:14:16,151 WARNING MsgRouterThr:2963634 [router.py:message_loop():75] message_loop has been closed

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/files/config.yaml

@@ -0,0 +1,242 @@
+_wandb:
+    value:
+        cli_version: 0.19.1
+        m: []
+        python_version: 3.8.20
+        t:
+            "1":
+                - 1
+                - 5
+                - 41
+                - 49
+                - 50
+                - 53
+                - 55
+                - 83
+            "2":
+                - 1
+                - 5
+                - 41
+                - 49
+                - 50
+                - 53
+                - 55
+                - 83
+            "3":
+                - 13
+                - 15
+                - 16
+                - 23
+                - 55
+                - 61
+            "4": 3.8.20
+            "5": 0.19.1
+            "8":
+                - 5
+            "12": 0.19.1
+            "13": linux-x86_64
+checkpoint:
+    value:
+        save_ckpt: true
+        save_last_ckpt: true
+        save_last_snapshot: false
+        topk:
+            format_str: epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt
+            k: 1
+            mode: max
+            monitor_key: test_mean_score
+dataloader:
+    value:
+        batch_size: 128
+        num_workers: 8
+        persistent_workers: false
+        pin_memory: true
+        shuffle: true
+dataset_obs_steps:
+    value: 2
+ema:
+    value:
+        _target_: diffusion_policy_3d.model.diffusion.ema_model.EMAModel
+        inv_gamma: 1
+        max_value: 0.9999
+        min_value: 0
+        power: 0.75
+        update_after_step: 0
+exp_name:
+    value: metaworld_button-press-dp3-1218
+horizon:
+    value: 4
+keypoint_visible_rate:
+    value: 1
+logging:
+    value:
+        group: metaworld_button-press-dp3-1218
+        id: null
+        mode: online
+        name: "0"
+        project: dp3
+        resume: true
+        tags:
+            - dp3
+multi_run:
+    value:
+        run_dir: data/outputs/2024.12.18/20.53.17_train_dp3_button-press
+        wandb_name_base: 2024.12.18-20.53.17_train_dp3_button-press
+n_action_steps:
+    value: 3
+n_latency_steps:
+    value: 0
+n_obs_steps:
+    value: 2
+name:
+    value: train_dp3
+obs_as_global_cond:
+    value: true
+optimizer:
+    value:
+        _target_: torch.optim.AdamW
+        betas:
+            - 0.95
+            - 0.999
+        eps: 1e-08
+        lr: 0.0001
+        weight_decay: 1e-06
+output_dir:
+    value: /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0
+policy:
+    value:
+        _target_: diffusion_policy_3d.policy.dp3.DP3
+        condition_type: film
+        crop_shape:
+            - 80
+            - 80
+        diffusion_step_embed_dim: 128
+        down_dims:
+            - 512
+            - 1024
+            - 2048
+        encoder_output_dim: 64
+        horizon: 4
+        kernel_size: 5
+        n_action_steps: 3
+        n_groups: 8
+        n_obs_steps: 2
+        noise_scheduler:
+            _target_: diffusers.schedulers.scheduling_ddim.DDIMScheduler
+            beta_end: 0.02
+            beta_schedule: squaredcos_cap_v2
+            beta_start: 0.0001
+            clip_sample: true
+            num_train_timesteps: 100
+            prediction_type: sample
+            set_alpha_to_one: true
+            steps_offset: 0
+        num_inference_steps: 10
+        obs_as_global_cond: true
+        pointcloud_encoder_cfg:
+            final_norm: layernorm
+            in_channels: 3
+            normal_channel: false
+            out_channels: 64
+            use_layernorm: true
+        pointnet_type: pointnet
+        shape_meta:
+            action:
+                shape:
+                    - 4
+            obs:
+                agent_pos:
+                    shape:
+                        - 9
+                    type: low_dim
+                point_cloud:
+                    shape:
+                        - 512
+                        - 3
+                    type: point_cloud
+        use_down_condition: true
+        use_mid_condition: true
+        use_pc_color: false
+        use_point_crop: true
+        use_up_condition: true
+shape_meta:
+    value:
+        action:
+            shape:
+                - 4
+        obs:
+            agent_pos:
+                shape:
+                    - 9
+                type: low_dim
+            point_cloud:
+                shape:
+                    - 512
+                    - 3
+                type: point_cloud
+task:
+    value:
+        dataset:
+            _target_: diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset
+            horizon: 4
+            max_train_episodes: 90
+            pad_after: 2
+            pad_before: 1
+            seed: 42
+            val_ratio: 0.02
+            zarr_path: /data/haojun/datasets/3d-dp/metaworld_button-press_expert.zarr
+        env_runner:
+            _target_: diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner
+            device: cuda:0
+            eval_episodes: 20
+            fps: 10
+            n_action_steps: 3
+            n_envs: null
+            n_obs_steps: 2
+            n_test: null
+            n_train: null
+            task_name: button-press
+            use_point_crop: true
+        name: button-press
+        shape_meta:
+            action:
+                shape:
+                    - 4
+            obs:
+                agent_pos:
+                    shape:
+                        - 9
+                    type: low_dim
+                point_cloud:
+                    shape:
+                        - 512
+                        - 3
+                    type: point_cloud
+        task_name: train_dp3
+task_name:
+    value: button-press
+training:
+    value:
+        checkpoint_every: 200
+        debug: false
+        device: cuda:0
+        gradient_accumulate_every: 1
+        lr_scheduler: cosine
+        lr_warmup_steps: 500
+        max_train_steps: null
+        max_val_steps: null
+        num_epochs: 3000
+        resume: true
+        rollout_every: 200
+        sample_every: 5
+        seed: 0
+        tqdm_interval_sec: 1
+        use_ema: true
+        val_every: 1
+val_dataloader:
+    value:
+        batch_size: 128
+        num_workers: 8
+        persistent_workers: false
+        pin_memory: true
+        shuffle: false

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/files/output.log

@@ -0,0 +1,46 @@
+                                                                                                                                                                                                       
+test_mean_score: 0.4
+mean_durations: 4.833745044376701
+std_durations: 0.014437844149605538
+test_mean_score: 1.0
+mean_durations: 4.84048391059041
+std_durations: 0.0033734295193179713
+test_mean_score: 1.0
+mean_durations: 4.839137175679207
+std_durations: 0.004018517936538068
+test_mean_score: 1.0
+mean_durations: 4.839165411889553
+std_durations: 0.004146314884051499
+test_mean_score: 1.0
+mean_durations: 4.837258285377175
+std_durations: 0.006565857215925964
+test_mean_score: 1.0
+mean_durations: 4.842524204682559
+std_durations: 0.005180089403291193
+test_mean_score: 1.0
+mean_durations: 4.845435710251332
+std_durations: 0.006345857496918728
+test_mean_score: 1.0
+mean_durations: 4.842679240740836
+std_durations: 0.0038064866747523885
+test_mean_score: 1.0
+mean_durations: 4.84310255870223
+std_durations: 0.004493370506277046
+test_mean_score: 1.0
+mean_durations: 4.8492849569767715
+std_durations: 0.008987950193715377
+test_mean_score: 1.0
+mean_durations: 4.850435198191553
+std_durations: 0.004243175674497617
+test_mean_score: 1.0
+mean_durations: 4.8532873706892135
+std_durations: 0.002826539924630156
+test_mean_score: 1.0
+mean_durations: 4.842780781537295
+std_durations: 0.005072163727256539
+test_mean_score: 1.0
+mean_durations: 4.842594298813492
+std_durations: 0.003441671512901404
+test_mean_score: 1.0
+mean_durations: 4.843653115443885
+std_durations: 0.00362016760922954

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/files/requirements.txt

@@ -0,0 +1,175 @@
+diffusion_policy_3d==0.0.0
+certifi==2024.12.14
+PyYAML==6.0.2
+dash-table==5.0.0
+dm_control==1.0.23
+einops==0.4.1
+ptyprocess==0.7.0
+jsonschema==4.23.0
+prompt_toolkit==3.0.48
+nbformat==5.10.4
+decorator==4.4.2
+importlib_metadata==8.5.0
+numcodecs==0.12.1
+requests==2.32.3
+ipywidgets==8.1.5
+urllib3==2.2.3
+platformdirs==4.3.6
+imageio-ffmpeg==0.5.1
+dash==2.18.2
+Cython==0.29.35
+mujoco-py==2.1.2.14
+mpmath==1.3.0
+typing_extensions==4.12.2
+dm-tree==0.1.8
+imageio==2.35.1
+dill==0.3.5.1
+joblib==1.4.2
+attrs==24.3.0
+torchvision==0.19.1+cu121
+kiwisolver==1.4.7
+numpy==1.23.5
+pillow==10.2.0
+nvidia-cuda-cupti-cu12==12.1.105
+pip==24.2
+llvmlite==0.39.1
+tenacity==9.0.0
+scipy==1.10.1
+charset-normalizer==3.4.0
+matplotlib==3.7.5
+jupyterlab_widgets==3.0.13
+patchelf==0.17.2.0
+wcwidth==0.2.13
+pexpect==4.9.0
+plotly==5.24.1
+ipdb==0.13.13
+referencing==0.35.1
+mujoco==3.2.3
+portalocker==3.0.0
+jedi==0.19.2
+Jinja2==3.1.3
+MarkupSafe==2.1.5
+nvidia-nvtx-cu12==12.1.105
+python-dateutil==2.9.0.post0
+packaging==24.2
+click==8.1.7
+pycparser==2.22
+blinker==1.8.2
+sympy==1.13.1
+Werkzeug==3.0.6
+six==1.17.0
+stack-data==0.6.3
+gpustat==1.1.1
+threadpoolctl==3.5.0
+traitlets==5.14.3
+annotated-types==0.7.0
+nvidia-nvjitlink-cu12==12.1.105
+zarr==2.12.0
+pandas==2.0.3
+wheel==0.44.0
+nvidia-nccl-cu12==2.20.5
+comm==0.2.2
+proglog==0.1.10
+parso==0.8.4
+open3d==0.18.0
+nvidia-cusparse-cu12==12.1.0.106
+antlr4-python3-runtime==4.9.3
+fvcore==0.1.5.post20221221
+natsort==8.4.0
+cloudpickle==3.1.0
+ConfigArgParse==1.7
+fonttools==4.55.3
+etils==1.3.0
+torchaudio==2.4.1+cu121
+pickleshare==0.7.5
+eval_type_backport==0.2.0
+executing==2.1.0
+addict==2.4.0
+protobuf==5.29.1
+setproctitle==1.3.4
+Flask==3.0.3
+filelock==3.13.1
+regex==2024.11.6
+PyOpenGL==3.1.7
+nvidia-curand-cu12==10.3.2.106
+matplotlib-inline==0.1.7
+transforms3d==0.4.2
+pydantic==2.10.3
+cffi==1.17.1
+rpds-py==0.20.1
+diffusers==0.11.1
+docker-pycreds==0.4.0
+pyparsing==3.1.4
+absl-py==2.1.0
+nvidia-cufft-cu12==11.0.2.54
+contourpy==1.1.1
+fastjsonschema==2.21.1
+pkgutil_resolve_name==1.3.10
+dash-html-components==2.0.0
+asciitree==0.3.3
+fasteners==0.19
+glfw==2.8.0
+omegaconf==2.3.0
+asttokens==3.0.0
+moviepy==1.0.3
+GitPython==3.1.43
+opencv-python==4.10.0.84
+zipp==3.20.2
+tabulate==0.9.0
+iopath==0.1.10
+labmaze==1.0.6
+numba==0.56.4
+nvidia-cuda-runtime-cu12==12.1.105
+dm-env==1.6
+cycler==0.12.1
+dash-core-components==2.0.0
+networkx==3.0
+Pygments==2.18.0
+retrying==1.3.4
+jsonschema-specifications==2023.12.1
+setuptools==59.5.0
+av==12.3.0
+itsdangerous==2.2.0
+wandb==0.19.1
+nest-asyncio==1.6.0
+triton==3.0.0
+nvidia-ml-py==12.560.30
+nvidia-cuda-nvrtc-cu12==12.1.105
+tomli==2.2.1
+tqdm==4.67.1
+termcolor==2.4.0
+hydra-core==1.2.0
+smmap==5.0.1
+ipython==8.12.3
+pyquaternion==0.9.9
+idna==3.10
+yacs==0.1.8
+tzdata==2024.2
+torch==2.4.1+cu121
+widgetsnbextension==4.0.13
+scikit-learn==1.3.2
+sapien==2.2.1
+nvidia-cublas-cu12==12.1.3.1
+huggingface-hub==0.21.0
+gitdb==4.0.11
+importlib_resources==6.4.5
+pydantic_core==2.27.1
+jupyter_core==5.7.2
+sentry-sdk==2.19.2
+nvidia-cudnn-cu12==9.1.0.70
+fsspec==2024.2.0
+psutil==6.1.0
+pytz==2024.2
+backcall==0.2.0
+nvidia-cusolver-cu12==11.4.5.107
+lxml==5.3.0
+blessed==1.20.0
+pure_eval==0.2.3
+diffusion_policy_3d==0.0.0
+dexart==0.1.0
+gym==0.21.0
+metaworld==0.1.0
+mj-envs==1.0.0
+mjrl==1.0.0
+pytorch3d==0.7.6
+diffusion_policy_3d==0.0.0

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/files/wandb-metadata.json

@@ -0,0 +1,95 @@
+{
+  "os":  "Linux-5.15.0-117-generic-x86_64-with-glibc2.17",
+  "python":  "CPython 3.8.20",
+  "startedAt":  "2024-12-18T12:53:19.199025Z",
+  "args":  [
+    "--config-name=dp3.yaml",
+    "task=metaworld_button-press",
+    "hydra.run.dir=data/outputs/metaworld_button-press-dp3-1218_seed0",
+    "training.debug=False",
+    "training.seed=0",
+    "training.device=cuda:0",
+    "exp_name=metaworld_button-press-dp3-1218",
+    "logging.mode=online",
+    "checkpoint.save_ckpt=True"
+  ],
+  "program":  "train.py",
+  "codePath":  "3D-Diffusion-Policy/train.py",
+  "git":  {
+    "remote":  "git@github.com:YanjieZe/3D-Diffusion-Policy.git",
+    "commit":  "3153fb32c17308fc828015156480b77f6b2e666a"
+  },
+  "email":  "chenhaojun001@gmail.com",
+  "root":  "/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0",
+  "host":  "Server-3090",
+  "executable":  "/home/chenhaojun/miniconda3/envs/dp3/bin/python",
+  "codePathLocal":  "train.py",
+  "cpu_count":  128,
+  "cpu_count_logical":  256,
+  "gpu":  "NVIDIA GeForce RTX 3090",
+  "gpu_count":  8,
+  "disk":  {
+    "/":  {
+      "total":  "943412031488",
+      "used":  "779944194048"
+    }
+  },
+  "memory":  {
+    "total":  "1081980669952"
+  },
+  "cpu":  {
+    "count":  128,
+    "countLogical":  256
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    },
+    {
+      "name":  "NVIDIA GeForce RTX 3090",
+      "memoryTotal":  "25769803776",
+      "cudaCores":  10496,
+      "architecture":  "Ampere"
+    }
+  ],
+  "cudaVersion":  "12.5"
+}

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/files/wandb-summary.json

@@ -0,0 +1 @@
+{"global_step":44999,"lr":0,"mean_traj_rewards":6207.491033874034,"SR_test_L5":1,"_wandb":{"runtime":12056},"_step":44999,"train_action_mse_error":5.829917881783331e-06,"SR_test_L3":1,"mean_success_rates":1,"train_loss":3.8665099092819827e-07,"epoch":2999,"mean_durations":4.843653115443885,"_runtime":12056.951864801,"bc_loss":1.9728383904293878e-07,"std_durations":0.00362016760922954,"_timestamp":1.7345384561434207e+09,"test_mean_score":1}

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug-internal.log

@@ -0,0 +1,16 @@
+{"time":"2024-12-18T20:53:19.200306453+08:00","level":"INFO","msg":"using version","core version":"0.19.1"}
+{"time":"2024-12-18T20:53:19.200319883+08:00","level":"INFO","msg":"created symlink","path":"/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug-core.log"}
+{"time":"2024-12-18T20:53:19.41309207+08:00","level":"INFO","msg":"created new stream","id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413148729+08:00","level":"INFO","msg":"stream: started","id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413198479+08:00","level":"INFO","msg":"writer: Do: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413237219+08:00","level":"INFO","msg":"sender: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:19.413334339+08:00","level":"INFO","msg":"handler: started","stream_id":"rdsa8ydn"}
+{"time":"2024-12-18T20:53:21.942657185+08:00","level":"INFO","msg":"Starting system monitor"}
+{"time":"2024-12-19T00:14:16.150854271+08:00","level":"INFO","msg":"stream: closing","id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:16.150900131+08:00","level":"INFO","msg":"Stopping system monitor"}
+{"time":"2024-12-19T00:14:16.152117904+08:00","level":"INFO","msg":"Stopped system monitor"}
+{"time":"2024-12-19T00:14:19.106555074+08:00","level":"INFO","msg":"fileTransfer: Close: file transfer manager closed"}
+{"time":"2024-12-19T00:14:20.252272164+08:00","level":"INFO","msg":"handler: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252345544+08:00","level":"INFO","msg":"sender: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252336774+08:00","level":"INFO","msg":"writer: Close: closed","stream_id":"rdsa8ydn"}
+{"time":"2024-12-19T00:14:20.252507633+08:00","level":"INFO","msg":"stream: closed","id":"rdsa8ydn"}

metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug.log

@@ -0,0 +1,24 @@
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Current SDK version is 0.19.1
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Configure stats pid to 2963634
+2024-12-18 20:53:19,169 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/.config/wandb/settings
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/wandb/settings
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_setup.py:_flush():68] Loading settings from environment variables
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:_log_setup():528] Logging user logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug.log
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:_log_setup():529] Logging internal logs to /home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0/wandb/run-20241218_205319-rdsa8ydn/logs/debug-internal.log
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():644] calling init triggers
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():650] wandb.init called with sweep_config: {}
+config: {'task': {'name': 'button-press', 'task_name': 'train_dp3', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'env_runner': {'_target_': 'diffusion_policy_3d.env_runner.metaworld_runner.MetaworldRunner', 'eval_episodes': 20, 'n_obs_steps': 2, 'n_action_steps': 3, 'fps': 10, 'n_envs': None, 'n_train': None, 'n_test': None, 'task_name': 'button-press', 'device': 'cuda:0', 'use_point_crop': True}, 'dataset': {'_target_': 'diffusion_policy_3d.dataset.metaworld_dataset.MetaworldDataset', 'zarr_path': '/data/haojun/datasets/3d-dp/metaworld_button-press_expert.zarr', 'horizon': 4, 'pad_before': 1, 'pad_after': 2, 'seed': 42, 'val_ratio': 0.02, 'max_train_episodes': 90}}, 'name': 'train_dp3', 'task_name': 'button-press', 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'exp_name': 'metaworld_button-press-dp3-1218', 'horizon': 4, 'n_obs_steps': 2, 'n_action_steps': 3, 'n_latency_steps': 0, 'dataset_obs_steps': 2, 'keypoint_visible_rate': 1.0, 'obs_as_global_cond': True, 'policy': {'_target_': 'diffusion_policy_3d.policy.dp3.DP3', 'use_point_crop': True, 'condition_type': 'film', 'use_down_condition': True, 'use_mid_condition': True, 'use_up_condition': True, 'diffusion_step_embed_dim': 128, 'down_dims': [512, 1024, 2048], 'crop_shape': [80, 80], 'encoder_output_dim': 64, 'horizon': 4, 'kernel_size': 5, 'n_action_steps': 3, 'n_groups': 8, 'n_obs_steps': 2, 'noise_scheduler': {'_target_': 'diffusers.schedulers.scheduling_ddim.DDIMScheduler', 'num_train_timesteps': 100, 'beta_start': 0.0001, 'beta_end': 0.02, 'beta_schedule': 'squaredcos_cap_v2', 'clip_sample': True, 'set_alpha_to_one': True, 'steps_offset': 0, 'prediction_type': 'sample'}, 'num_inference_steps': 10, 'obs_as_global_cond': True, 'shape_meta': {'obs': {'point_cloud': {'shape': [512, 3], 'type': 'point_cloud'}, 'agent_pos': {'shape': [9], 'type': 'low_dim'}}, 'action': {'shape': [4]}}, 'use_pc_color': False, 'pointnet_type': 'pointnet', 'pointcloud_encoder_cfg': {'in_channels': 3, 'out_channels': 64, 'use_layernorm': True, 'final_norm': 'layernorm', 'normal_channel': False}}, 'ema': {'_target_': 'diffusion_policy_3d.model.diffusion.ema_model.EMAModel', 'update_after_step': 0, 'inv_gamma': 1.0, 'power': 0.75, 'min_value': 0.0, 'max_value': 0.9999}, 'dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': True, 'pin_memory': True, 'persistent_workers': False}, 'val_dataloader': {'batch_size': 128, 'num_workers': 8, 'shuffle': False, 'pin_memory': True, 'persistent_workers': False}, 'optimizer': {'_target_': 'torch.optim.AdamW', 'lr': 0.0001, 'betas': [0.95, 0.999], 'eps': 1e-08, 'weight_decay': 1e-06}, 'training': {'device': 'cuda:0', 'seed': 0, 'debug': False, 'resume': True, 'lr_scheduler': 'cosine', 'lr_warmup_steps': 500, 'num_epochs': 3000, 'gradient_accumulate_every': 1, 'use_ema': True, 'rollout_every': 200, 'checkpoint_every': 200, 'val_every': 1, 'sample_every': 5, 'max_train_steps': None, 'max_val_steps': None, 'tqdm_interval_sec': 1.0}, 'logging': {'group': 'metaworld_button-press-dp3-1218', 'id': None, 'mode': 'online', 'name': '0', 'project': 'dp3', 'resume': True, 'tags': ['dp3']}, 'checkpoint': {'save_ckpt': True, 'topk': {'monitor_key': 'test_mean_score', 'mode': 'max', 'k': 1, 'format_str': 'epoch={epoch:04d}-test_mean_score={test_mean_score:.3f}.ckpt'}, 'save_last_ckpt': True, 'save_last_snapshot': False}, 'multi_run': {'run_dir': 'data/outputs/2024.12.18/20.53.17_train_dp3_button-press', 'wandb_name_base': '2024.12.18-20.53.17_train_dp3_button-press'}}
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():680] starting backend
+2024-12-18 20:53:19,170 INFO    MainThread:2963634 [wandb_init.py:init():684] sending inform_init request
+2024-12-18 20:53:19,198 INFO    MainThread:2963634 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2024-12-18 20:53:19,198 INFO    MainThread:2963634 [wandb_init.py:init():697] backend started and connected
+2024-12-18 20:53:19,203 INFO    MainThread:2963634 [wandb_init.py:init():790] updated telemetry
+2024-12-18 20:53:19,347 INFO    MainThread:2963634 [wandb_init.py:init():822] communicating run to backend with 90.0 second timeout
+2024-12-18 20:53:21,937 INFO    MainThread:2963634 [wandb_init.py:init():874] starting run threads in backend
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_console_start():2374] atexit reg
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2224] redirect: wrap_raw
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2289] Wrapping output streams.
+2024-12-18 20:53:22,052 INFO    MainThread:2963634 [wandb_run.py:_redirect():2314] Redirects installed.
+2024-12-18 20:53:22,054 INFO    MainThread:2963634 [wandb_init.py:init():916] run started, returning control to user process
+2024-12-18 20:53:22,055 INFO    MainThread:2963634 [wandb_run.py:_config_callback():1279] config_cb None None {'output_dir': '/home/chenhaojun/3D-Diffusion-Policy/3D-Diffusion-Policy/data/outputs/metaworld_button-press-dp3-1218_seed0'}
+2024-12-19 00:14:16,151 WARNING MsgRouterThr:2963634 [router.py:message_loop():75] message_loop has been closed