| INFO 2025-10-30 16:50:08 db_utils.py:102 [1m[34mLogs will be synced with wandb.[0m | |
| INFO 2025-10-30 16:50:08 db_utils.py:103 Track this run --> [1m[33mhttps://wandb.ai/era/lerobot/runs/zyne4bj1[0m | |
| INFO 2025-10-30 16:50:08 ot_train.py:183 Creating dataset | |
| info.json: 3.23kB [00:00, 6.26MB/s], ?it/s] | |
| stats.json: 13.0kB [00:00, 6.69MB/s]quet: 0% 0.00/61.2k [00:00<?, ?B/s] | |
| meta/tasks.parquet: 100% 2.18k/2.18k [00:00<00:00, 2.65kB/s] | |
| meta/episodes/chunk-000/file-000.parquet: 100% 61.2k/61.2k [00:00<00:00, 63.8kB/s] | |
| Fetching 4 files: 100% 4/4 [00:01<00:00, 3.25it/s]2.66kB/s] | |
| README.md: 3.71kB [00:00, 8.68MB/s], ?it/s] | |
| .gitattributes: 2.46kB [00:00, 6.30MB/s](…): 0% 0.00/10.3M [00:00<?, ?B/s] | |
| data/chunk-000/file-000.parquet: 100% 109k/109k [00:00<00:00, 462kB/s] ?B/s] | |
| videos/observation.images.side/chunk-000(…): 100% 8.45M/8.45M [00:00<00:00, 8.55MB/s] | |
| videos/observation.images.front/chunk-00(…): 100% 10.3M/10.3M [00:01<00:00, 10.2MB/s] | |
| Fetching 9 files: 100% 9/9 [00:01<00:00, 7.31it/s]0.3M/10.3M [00:01<00:00, 10.2MB/s] | |
| INFO 2025-10-30 16:50:11 ot_train.py:202 Creating policy | |
| Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth | |
| 100% 44.7M/44.7M [00:00<00:00, 175MB/s] | |
| INFO 2025-10-30 16:50:13 ot_train.py:247 Creating optimizer and scheduler | |
| INFO 2025-10-30 16:50:13 ot_train.py:259 [1m[33mOutput dir:[0m outputs/train/test_touch_x | |
| INFO 2025-10-30 16:50:13 ot_train.py:262 cfg.steps=100000 (100K) | |
| INFO 2025-10-30 16:50:13 ot_train.py:263 dataset.num_frames=2072 (2K) | |
| INFO 2025-10-30 16:50:13 ot_train.py:264 dataset.num_episodes=7 | |
| INFO 2025-10-30 16:50:13 ot_train.py:267 Effective batch size: 8 x 1 = 8 | |
| INFO 2025-10-30 16:50:13 ot_train.py:268 num_learnable_params=51597190 (52M) | |
| INFO 2025-10-30 16:50:13 ot_train.py:269 num_total_params=51597190 (52M) | |
| /usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py:626: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. | |
| warnings.warn( | |
| INFO 2025-10-30 16:50:13 ot_train.py:324 Start offline training on a fixed dataset | |
| INFO 2025-10-30 16:52:02 ot_train.py:351 step:200 smpl:2K ep:5 epch:0.77 loss:6.621 grdn:151.412 lr:1.0e-05 updt_s:0.523 data_s:0.023 | |
| INFO 2025-10-30 16:53:49 ot_train.py:351 step:400 smpl:3K ep:11 epch:1.54 loss:2.841 grdn:83.299 lr:1.0e-05 updt_s:0.515 data_s:0.021 | |
| INFO 2025-10-30 16:55:37 ot_train.py:351 step:600 smpl:5K ep:16 epch:2.32 loss:2.392 grdn:72.401 lr:1.0e-05 updt_s:0.519 data_s:0.019 | |
| INFO 2025-10-30 16:57:26 ot_train.py:351 step:800 smpl:6K ep:22 epch:3.09 loss:2.110 grdn:68.692 lr:1.0e-05 updt_s:0.521 data_s:0.022 | |
| INFO 2025-10-30 16:59:12 ot_train.py:351 step:1K smpl:8K ep:27 epch:3.86 loss:1.832 grdn:65.134 lr:1.0e-05 updt_s:0.521 data_s:0.007 | |
| INFO 2025-10-30 17:01:00 ot_train.py:351 step:1K smpl:10K ep:32 epch:4.63 loss:1.625 grdn:61.554 lr:1.0e-05 updt_s:0.521 data_s:0.019 | |
| INFO 2025-10-30 17:02:48 ot_train.py:351 step:1K smpl:11K ep:38 epch:5.41 loss:1.493 grdn:59.241 lr:1.0e-05 updt_s:0.522 data_s:0.018 | |
| INFO 2025-10-30 17:04:36 ot_train.py:351 step:2K smpl:13K ep:43 epch:6.18 loss:1.355 grdn:56.801 lr:1.0e-05 updt_s:0.521 data_s:0.018 | |
| INFO 2025-10-30 17:06:22 ot_train.py:351 step:2K smpl:14K ep:49 epch:6.95 loss:1.218 grdn:52.597 lr:1.0e-05 updt_s:0.522 data_s:0.007 | |
| INFO 2025-10-30 17:08:10 ot_train.py:351 step:2K smpl:16K ep:54 epch:7.72 loss:1.109 grdn:51.568 lr:1.0e-05 updt_s:0.521 data_s:0.019 | |
| INFO 2025-10-30 17:09:58 ot_train.py:351 step:2K smpl:18K ep:59 epch:8.49 loss:0.996 grdn:48.094 lr:1.0e-05 updt_s:0.521 data_s:0.021 | |
| INFO 2025-10-30 17:11:46 ot_train.py:351 step:2K smpl:19K ep:65 epch:9.27 loss:0.905 grdn:46.796 lr:1.0e-05 updt_s:0.521 data_s:0.018 | |
| INFO 2025-10-30 17:13:36 ot_train.py:351 step:3K smpl:21K ep:70 epch:10.04 loss:0.808 grdn:43.320 lr:1.0e-05 updt_s:0.522 data_s:0.024 | |
| INFO 2025-10-30 17:15:22 ot_train.py:351 step:3K smpl:22K ep:76 epch:10.81 loss:0.734 grdn:40.908 lr:1.0e-05 updt_s:0.524 data_s:0.008 | |
| INFO 2025-10-30 17:17:10 ot_train.py:351 step:3K smpl:24K ep:81 epch:11.58 loss:0.658 grdn:39.269 lr:1.0e-05 updt_s:0.522 data_s:0.018 | |
| INFO 2025-10-30 17:18:59 ot_train.py:351 step:3K smpl:26K ep:86 epch:12.36 loss:0.601 grdn:37.722 lr:1.0e-05 updt_s:0.520 data_s:0.021 | |
| INFO 2025-10-30 17:20:47 ot_train.py:351 step:3K smpl:27K ep:92 epch:13.13 loss:0.535 grdn:35.455 lr:1.0e-05 updt_s:0.522 data_s:0.018 | |
| INFO 2025-10-30 17:22:33 ot_train.py:351 step:4K smpl:29K ep:97 epch:13.90 loss:0.499 grdn:34.400 lr:1.0e-05 updt_s:0.521 data_s:0.007 | |
| INFO 2025-10-30 17:24:22 ot_train.py:351 step:4K smpl:30K ep:103 epch:14.67 loss:0.452 grdn:32.424 lr:1.0e-05 updt_s:0.521 data_s:0.021 | |
| INFO 2025-10-30 17:26:10 ot_train.py:351 step:4K smpl:32K ep:108 epch:15.44 loss:0.410 grdn:30.819 lr:1.0e-05 updt_s:0.521 data_s:0.019 | |
| INFO 2025-10-30 17:27:59 ot_train.py:351 step:4K smpl:34K ep:114 epch:16.22 loss:0.378 grdn:29.308 lr:1.0e-05 updt_s:0.521 data_s:0.024 | |
| INFO 2025-10-30 17:29:45 ot_train.py:351 step:4K smpl:35K ep:119 epch:16.99 loss:0.346 grdn:27.904 lr:1.0e-05 updt_s:0.521 data_s:0.007 | |
| INFO 2025-10-30 17:31:33 ot_train.py:351 step:5K smpl:37K ep:124 epch:17.76 loss:0.325 grdn:27.193 lr:1.0e-05 updt_s:0.521 data_s:0.019 | |
| INFO 2025-10-30 17:33:21 ot_train.py:351 step:5K smpl:38K ep:130 epch:18.53 loss:0.300 grdn:26.294 lr:1.0e-05 updt_s:0.522 data_s:0.019 | |
| INFO 2025-10-30 17:35:10 ot_train.py:351 step:5K smpl:40K ep:135 epch:19.31 loss:0.283 grdn:25.573 lr:1.0e-05 updt_s:0.521 data_s:0.021 | |
| INFO 2025-10-30 17:37:00 ot_train.py:351 step:5K smpl:42K ep:141 epch:20.08 loss:0.264 grdn:25.378 lr:1.0e-05 updt_s:0.524 data_s:0.026 | |
| INFO 2025-10-30 17:38:45 ot_train.py:351 step:5K smpl:43K ep:146 epch:20.85 loss:0.253 grdn:23.929 lr:1.0e-05 updt_s:0.520 data_s:0.007 | |
| INFO 2025-10-30 17:40:34 ot_train.py:351 step:6K smpl:45K ep:151 epch:21.62 loss:0.235 grdn:22.867 lr:1.0e-05 updt_s:0.521 data_s:0.023 | |
| INFO 2025-10-30 17:42:22 ot_train.py:351 step:6K smpl:46K ep:157 epch:22.39 loss:0.223 grdn:22.505 lr:1.0e-05 updt_s:0.520 data_s:0.019 | |
| INFO 2025-10-30 17:44:10 ot_train.py:351 step:6K smpl:48K ep:162 epch:23.17 loss:0.214 grdn:22.232 lr:1.0e-05 updt_s:0.521 data_s:0.018 | |
| INFO 2025-10-30 17:45:56 ot_train.py:351 step:6K smpl:50K ep:168 epch:23.94 loss:0.205 grdn:21.189 lr:1.0e-05 updt_s:0.520 data_s:0.007 | |
| INFO 2025-10-30 17:47:45 ot_train.py:351 step:6K smpl:51K ep:173 epch:24.71 loss:0.196 grdn:20.942 lr:1.0e-05 updt_s:0.521 data_s:0.023 | |
| INFO 2025-10-30 17:49:33 ot_train.py:351 step:7K smpl:53K ep:178 epch:25.48 loss:0.189 grdn:20.612 lr:1.0e-05 updt_s:0.521 data_s:0.020 | |
| INFO 2025-10-30 17:51:21 ot_train.py:351 step:7K smpl:54K ep:184 epch:26.25 loss:0.181 grdn:20.462 lr:1.0e-05 updt_s:0.522 data_s:0.018 | |
| INFO 2025-10-30 17:53:09 ot_train.py:351 step:7K smpl:56K ep:189 epch:27.03 loss:0.176 grdn:19.728 lr:1.0e-05 updt_s:0.521 data_s:0.018 | |
| INFO 2025-10-30 17:54:55 ot_train.py:351 step:7K smpl:58K ep:195 epch:27.80 loss:0.172 grdn:19.352 lr:1.0e-05 updt_s:0.520 data_s:0.007 | |
| INFO 2025-10-30 17:56:43 ot_train.py:351 step:7K smpl:59K ep:200 epch:28.57 loss:0.164 grdn:18.971 lr:1.0e-05 updt_s:0.521 data_s:0.018 | |
| INFO 2025-10-30 17:58:32 ot_train.py:351 step:8K smpl:61K ep:205 epch:29.34 loss:0.160 grdn:18.534 lr:1.0e-05 updt_s:0.527 data_s:0.019 | |
| INFO 2025-10-30 18:00:21 ot_train.py:351 step:8K smpl:62K ep:211 epch:30.12 loss:0.153 grdn:17.849 lr:1.0e-05 updt_s:0.521 data_s:0.020 | |
| INFO 2025-10-30 18:02:06 ot_train.py:351 step:8K smpl:64K ep:216 epch:30.89 loss:0.150 grdn:18.069 lr:1.0e-05 updt_s:0.520 data_s:0.007 | |
| Traceback (most recent call last): | |
| File "/content/lerobot/src/lerobot/scripts/lerobot_train.py", line 448, in <module> | |
| main() | |
| File "/content/lerobot/src/lerobot/scripts/lerobot_train.py", line 444, in main | |
| train() | |
| File "/content/lerobot/src/lerobot/configs/parser.py", line 233, in wrapper_inner | |
| response = fn(cfg, *args, **kwargs) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/content/lerobot/src/lerobot/scripts/lerobot_train.py", line 332, in train | |
| train_tracker, output_dict = update_policy( | |
| ^^^^^^^^^^^^^^ | |
| File "/content/lerobot/src/lerobot/scripts/lerobot_train.py", line 119, in update_policy | |
| train_metrics.loss = loss.item() | |
| ^^^^^^^^^^^ | |
| KeyboardInterrupt | |