| [2026-07-19 07:37:34,910][fastwam.models.wan22.helpers.loader][INFO] - Loading Wan2.2-TI2V-5B components... |
| [2026-07-19 07:43:33,777][fastwam.models.wan22.helpers.loader][INFO] - Skipping pretrained text encoder/tokenizer load (`load_text_encoder=False`); training must provide cached `context/context_mask`. |
| [2026-07-19 07:44:41,973][fastwam.models.wan22.helpers.loader][INFO] - Finished loading Wan2.2-TI2V-5B components in 427.06 seconds. |
| [2026-07-19 07:45:44,137][fastwam.models.wan22.action_dit][INFO] - ActionDiT backbone payload policy: {'skip_prefixes': ['action_encoder.', 'head.'], 'alpha_scaling': True, 'interpolation': 'sequential_1d_linear_align_corners_true'} |
| [2026-07-19 07:45:44,603][fastwam.models.wan22.action_dit][INFO] - Loaded ActionDiT backbone from /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/derived/fastwam/actiondit/sha256-7bd65e1986accaaf2e8dd5e59b5e98ef348d7c2f1d88ab2df010eb954a441445/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt (keys=820; random_kept_prefixes=['action_encoder.', 'head.']). |
| [2026-07-19 07:45:44,949][fastwam.models.wan22.mot][INFO] - Initialized MoT with experts: ['video', 'action'], num_layers=30 |
| [2026-07-19 07:45:44,952][fastwam.models.wan22.mot][INFO] - Expert 'video': num_params=5.00 B |
| [2026-07-19 07:45:44,954][fastwam.models.wan22.mot][INFO] - Expert 'action': num_params=1.02 B |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,891][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:48,397][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpu0_zrl5y/test.c -o /tmp/tmpu0_zrl5y/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpw4vmqot9/test.c -o /tmp/tmpw4vmqot9/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp7ydupj13/test.c -o /tmp/tmp7ydupj13/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpv4_r1j7i/test.c -o /tmp/tmpv4_r1j7i/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpvlarjs01/test.c -o /tmp/tmpvlarjs01/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmph3mhqnk0/test.c -o /tmp/tmph3mhqnk0/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpl6mimau0/test.c -o /tmp/tmpl6mimau0/test.o |
| [2026-07-19 07:45:48,398][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp56i98wui/test.c -o /tmp/tmp56i98wui/test.o |
| [2026-07-19 07:45:48,492][root][INFO] - /usr/bin/gcc-11 /tmp/tmpw4vmqot9/test.o -laio -o /tmp/tmpw4vmqot9/a.out |
| [2026-07-19 07:45:48,492][root][INFO] - /usr/bin/gcc-11 /tmp/tmpvlarjs01/test.o -laio -o /tmp/tmpvlarjs01/a.out |
| [2026-07-19 07:45:48,493][root][INFO] - /usr/bin/gcc-11 /tmp/tmpv4_r1j7i/test.o -laio -o /tmp/tmpv4_r1j7i/a.out |
| [2026-07-19 07:45:48,493][root][INFO] - /usr/bin/gcc-11 /tmp/tmp7ydupj13/test.o -laio -o /tmp/tmp7ydupj13/a.out |
| [2026-07-19 07:45:48,493][root][INFO] - /usr/bin/gcc-11 /tmp/tmph3mhqnk0/test.o -laio -o /tmp/tmph3mhqnk0/a.out |
| [2026-07-19 07:45:48,493][root][INFO] - /usr/bin/gcc-11 /tmp/tmpu0_zrl5y/test.o -laio -o /tmp/tmpu0_zrl5y/a.out |
| [2026-07-19 07:45:48,493][root][INFO] - /usr/bin/gcc-11 /tmp/tmp56i98wui/test.o -laio -o /tmp/tmp56i98wui/a.out |
| [2026-07-19 07:45:48,492][root][INFO] - /usr/bin/gcc-11 /tmp/tmpl6mimau0/test.o -laio -o /tmp/tmpl6mimau0/a.out |
| [2026-07-19 07:45:48,577][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpnvacml8k/test.c -o /tmp/tmpnvacml8k/test.o |
| [2026-07-19 07:45:48,577][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmps1vwdcxw/test.c -o /tmp/tmps1vwdcxw/test.o |
| [2026-07-19 07:45:48,579][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpyj42xm2q/test.c -o /tmp/tmpyj42xm2q/test.o |
| [2026-07-19 07:45:48,579][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp5hymz2mb/test.c -o /tmp/tmp5hymz2mb/test.o |
| [2026-07-19 07:45:48,579][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp1pr_mzes/test.c -o /tmp/tmp1pr_mzes/test.o |
| [2026-07-19 07:45:48,606][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpx49xbchm/test.c -o /tmp/tmpx49xbchm/test.o |
| [2026-07-19 07:45:48,606][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpnwq2re2z/test.c -o /tmp/tmpnwq2re2z/test.o |
| [2026-07-19 07:45:48,606][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpj37jgohn/test.c -o /tmp/tmpj37jgohn/test.o |
| [2026-07-19 07:45:48,614][root][INFO] - /usr/bin/gcc-11 /tmp/tmpnvacml8k/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpnvacml8k/a.out |
| [2026-07-19 07:45:48,615][root][INFO] - /usr/bin/gcc-11 /tmp/tmps1vwdcxw/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmps1vwdcxw/a.out |
| [2026-07-19 07:45:48,637][root][INFO] - /usr/bin/gcc-11 /tmp/tmp1pr_mzes/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp1pr_mzes/a.out |
| [2026-07-19 07:45:48,637][root][INFO] - /usr/bin/gcc-11 /tmp/tmpyj42xm2q/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpyj42xm2q/a.out |
| [2026-07-19 07:45:48,637][root][INFO] - /usr/bin/gcc-11 /tmp/tmp5hymz2mb/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp5hymz2mb/a.out |
| [2026-07-19 07:45:48,658][root][INFO] - /usr/bin/gcc-11 /tmp/tmpx49xbchm/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpx49xbchm/a.out |
| [2026-07-19 07:45:48,657][root][INFO] - /usr/bin/gcc-11 /tmp/tmpnwq2re2z/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpnwq2re2z/a.out |
| [2026-07-19 07:45:48,658][root][INFO] - /usr/bin/gcc-11 /tmp/tmpj37jgohn/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpj37jgohn/a.out |
| [2026-07-19 07:45:48,694][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp2eviip_v/test.c -o /tmp/tmp2eviip_v/test.o |
| [2026-07-19 07:45:48,694][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmptpmfhefg/test.c -o /tmp/tmptpmfhefg/test.o |
| [2026-07-19 07:45:48,703][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpg346xs5u/test.c -o /tmp/tmpg346xs5u/test.o |
| [2026-07-19 07:45:48,703][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmps5lgx7w7/test.c -o /tmp/tmps5lgx7w7/test.o |
| [2026-07-19 07:45:48,703][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpyycnh81e/test.c -o /tmp/tmpyycnh81e/test.o |
| [2026-07-19 07:45:48,722][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpa52tlmyt/test.c -o /tmp/tmpa52tlmyt/test.o |
| [2026-07-19 07:45:48,722][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpnsuclexv/test.c -o /tmp/tmpnsuclexv/test.o |
| [2026-07-19 07:45:48,723][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpgq84s97a/test.c -o /tmp/tmpgq84s97a/test.o |
| [2026-07-19 07:45:48,727][root][INFO] - /usr/bin/gcc-11 /tmp/tmp2eviip_v/test.o -laio -o /tmp/tmp2eviip_v/a.out |
| [2026-07-19 07:45:48,727][root][INFO] - /usr/bin/gcc-11 /tmp/tmptpmfhefg/test.o -laio -o /tmp/tmptpmfhefg/a.out |
| [2026-07-19 07:45:48,747][root][INFO] - /usr/bin/gcc-11 /tmp/tmpg346xs5u/test.o -laio -o /tmp/tmpg346xs5u/a.out |
| [2026-07-19 07:45:48,748][root][INFO] - /usr/bin/gcc-11 /tmp/tmpyycnh81e/test.o -laio -o /tmp/tmpyycnh81e/a.out |
| [2026-07-19 07:45:48,748][root][INFO] - /usr/bin/gcc-11 /tmp/tmps5lgx7w7/test.o -laio -o /tmp/tmps5lgx7w7/a.out |
| [2026-07-19 07:45:48,769][root][INFO] - /usr/bin/gcc-11 /tmp/tmpgq84s97a/test.o -laio -o /tmp/tmpgq84s97a/a.out |
| [2026-07-19 07:45:48,770][root][INFO] - /usr/bin/gcc-11 /tmp/tmpnsuclexv/test.o -laio -o /tmp/tmpnsuclexv/a.out |
| [2026-07-19 07:45:48,770][root][INFO] - /usr/bin/gcc-11 /tmp/tmpa52tlmyt/test.o -laio -o /tmp/tmpa52tlmyt/a.out |
| [2026-07-19 07:45:54,822][fastwam.datasets.lerobot.robot_video_dataset][INFO] - Calculating dataset stats for normalization... |
| [2026-07-19 07:45:55,876][fastwam.trainer][INFO] - Accelerate training: distributed_type=DistributedType.DEEPSPEED zero_stage=2 world_size=8 process_index=0 cfg_mixed_precision=bf16 accelerator_mixed_precision=bf16 grad_accum=1 grad_clip=1.0000 |
| [2026-07-19 07:45:55,878][fastwam.trainer][INFO] - using accelerator.device=cuda:0 |
| [2026-07-19 07:46:05,285][fastwam.trainer][INFO] - Train/val dataset size: 41464/41464 |
| [2026-07-19 07:46:05,286][fastwam.trainer][INFO] - Setting DiT to train mode and freezing other model components. |
| [2026-07-19 07:46:05,304][fastwam.trainer][INFO] - Starting training with max_steps=1620. |
| [2026-07-19 07:53:57,899][fastwam.trainer][INFO] - [train] epoch=0 step=10/1620 loss=1.6926 loss_action=1.2874 loss_video=0.4052 lr=1.34e-05 speed=0.02 step/s, 2.71 samples/s eta=21:08:07 |
| [2026-07-19 08:01:09,960][fastwam.trainer][INFO] - [train] epoch=0 step=20/1620 loss=0.8043 loss_action=0.4747 loss_video=0.3296 lr=2.56e-05 speed=0.02 step/s, 2.83 samples/s eta=20:06:12 |
| [2026-07-19 08:08:22,659][fastwam.trainer][INFO] - [train] epoch=0 step=30/1620 loss=0.5874 loss_action=0.3160 loss_video=0.2714 lr=3.78e-05 speed=0.02 step/s, 2.87 samples/s eta=19:41:19 |
| [2026-07-19 08:15:33,818][fastwam.trainer][INFO] - [train] epoch=0 step=40/1620 loss=0.4502 loss_action=0.2023 loss_video=0.2480 lr=5.00e-05 speed=0.02 step/s, 2.90 samples/s eta=19:24:16 |
| [2026-07-19 08:22:46,516][fastwam.trainer][INFO] - [train] epoch=0 step=50/1620 loss=0.4311 loss_action=0.1886 loss_video=0.2424 lr=6.22e-05 speed=0.02 step/s, 2.91 samples/s eta=19:11:58 |
| [2026-07-19 08:29:56,983][fastwam.trainer][INFO] - [train] epoch=0 step=60/1620 loss=0.3786 loss_action=0.1667 loss_video=0.2118 lr=7.44e-05 speed=0.02 step/s, 2.92 samples/s eta=19:00:23 |
| [2026-07-19 08:37:07,348][fastwam.trainer][INFO] - [train] epoch=0 step=70/1620 loss=0.3227 loss_action=0.1000 loss_video=0.2227 lr=8.66e-05 speed=0.02 step/s, 2.93 samples/s eta=18:50:02 |
| [2026-07-19 08:44:18,777][fastwam.trainer][INFO] - [train] epoch=0 step=80/1620 loss=0.3187 loss_action=0.0867 loss_video=0.2320 lr=9.88e-05 speed=0.02 step/s, 2.93 samples/s eta=18:40:49 |
| [2026-07-19 08:51:28,865][fastwam.trainer][INFO] - [train] epoch=0 step=90/1620 loss=0.3030 loss_action=0.0816 loss_video=0.2215 lr=1.00e-04 speed=0.02 step/s, 2.94 samples/s eta=18:31:40 |
| [2026-07-19 08:58:37,401][fastwam.trainer][INFO] - [train] epoch=0 step=100/1620 loss=0.2817 loss_action=0.0729 loss_video=0.2088 lr=1.00e-04 speed=0.02 step/s, 2.94 samples/s eta=18:22:31 |
| [2026-07-19 09:05:50,365][fastwam.trainer][INFO] - [train] epoch=0 step=110/1620 loss=0.2644 loss_action=0.0746 loss_video=0.1898 lr=9.99e-05 speed=0.02 step/s, 2.94 samples/s eta=18:14:45 |
| [2026-07-19 09:13:01,929][fastwam.trainer][INFO] - [train] epoch=0 step=120/1620 loss=0.2542 loss_action=0.0627 loss_video=0.1915 lr=9.98e-05 speed=0.02 step/s, 2.94 samples/s eta=18:06:47 |
| [2026-07-19 09:20:14,674][fastwam.trainer][INFO] - [train] epoch=0 step=130/1620 loss=0.2393 loss_action=0.0555 loss_video=0.1838 lr=9.98e-05 speed=0.02 step/s, 2.95 samples/s eta=17:59:10 |
| [2026-07-19 09:27:28,170][fastwam.trainer][INFO] - [train] epoch=0 step=140/1620 loss=0.2519 loss_action=0.0502 loss_video=0.2016 lr=9.96e-05 speed=0.02 step/s, 2.95 samples/s eta=17:51:44 |
| [2026-07-19 09:34:39,923][fastwam.trainer][INFO] - [train] epoch=0 step=150/1620 loss=0.2413 loss_action=0.0365 loss_video=0.2048 lr=9.95e-05 speed=0.02 step/s, 2.95 samples/s eta=17:44:03 |
| [2026-07-19 09:41:51,255][fastwam.trainer][INFO] - [train] epoch=0 step=160/1620 loss=0.2028 loss_action=0.0489 loss_video=0.1539 lr=9.94e-05 speed=0.02 step/s, 2.95 samples/s eta=17:36:21 |
| [2026-07-19 09:49:03,015][fastwam.trainer][INFO] - [train] epoch=0 step=170/1620 loss=0.1954 loss_action=0.0412 loss_video=0.1542 lr=9.92e-05 speed=0.02 step/s, 2.95 samples/s eta=17:28:47 |
| [2026-07-19 09:56:14,265][fastwam.trainer][INFO] - [train] epoch=0 step=180/1620 loss=0.2303 loss_action=0.0422 loss_video=0.1881 lr=9.90e-05 speed=0.02 step/s, 2.95 samples/s eta=17:21:11 |
| [2026-07-19 10:03:25,703][fastwam.trainer][INFO] - [train] epoch=0 step=190/1620 loss=0.2052 loss_action=0.0468 loss_video=0.1584 lr=9.88e-05 speed=0.02 step/s, 2.95 samples/s eta=17:13:39 |
| [2026-07-19 10:10:37,959][fastwam.trainer][INFO] - [train] epoch=0 step=200/1620 loss=0.2324 loss_action=0.0528 loss_video=0.1797 lr=9.85e-05 speed=0.02 step/s, 2.95 samples/s eta=17:06:15 |
| [2026-07-19 10:17:53,504][fastwam.trainer][INFO] - [train] epoch=0 step=210/1620 loss=0.2256 loss_action=0.0346 loss_video=0.1910 lr=9.83e-05 speed=0.02 step/s, 2.95 samples/s eta=16:59:15 |
| [2026-07-19 10:25:09,394][fastwam.trainer][INFO] - [train] epoch=0 step=220/1620 loss=0.1786 loss_action=0.0320 loss_video=0.1467 lr=9.80e-05 speed=0.02 step/s, 2.95 samples/s eta=16:52:15 |
| [2026-07-19 10:32:24,781][fastwam.trainer][INFO] - [train] epoch=0 step=230/1620 loss=0.2229 loss_action=0.0295 loss_video=0.1934 lr=9.77e-05 speed=0.02 step/s, 2.95 samples/s eta=16:45:10 |
| [2026-07-19 10:39:40,823][fastwam.trainer][INFO] - [train] epoch=0 step=240/1620 loss=0.1989 loss_action=0.0276 loss_video=0.1713 lr=9.74e-05 speed=0.02 step/s, 2.95 samples/s eta=16:38:09 |
| [2026-07-19 10:46:57,916][fastwam.trainer][INFO] - [train] epoch=0 step=250/1620 loss=0.2325 loss_action=0.0335 loss_video=0.1990 lr=9.71e-05 speed=0.02 step/s, 2.95 samples/s eta=16:31:12 |
| [2026-07-19 10:54:11,458][fastwam.trainer][INFO] - [train] epoch=0 step=260/1620 loss=0.2296 loss_action=0.0426 loss_video=0.1870 lr=9.67e-05 speed=0.02 step/s, 2.95 samples/s eta=16:23:55 |
| [2026-07-19 11:01:31,535][fastwam.trainer][INFO] - [train] epoch=0 step=270/1620 loss=0.1848 loss_action=0.0251 loss_video=0.1598 lr=9.64e-05 speed=0.02 step/s, 2.95 samples/s eta=16:17:11 |
| [2026-07-19 11:08:51,117][fastwam.trainer][INFO] - [train] epoch=0 step=280/1620 loss=0.2047 loss_action=0.0309 loss_video=0.1738 lr=9.60e-05 speed=0.02 step/s, 2.95 samples/s eta=16:10:22 |
| [2026-07-19 11:16:08,674][fastwam.trainer][INFO] - [train] epoch=0 step=290/1620 loss=0.1914 loss_action=0.0252 loss_video=0.1661 lr=9.56e-05 speed=0.02 step/s, 2.95 samples/s eta=16:03:21 |
| [2026-07-19 11:23:26,974][fastwam.trainer][INFO] - [train] epoch=0 step=300/1620 loss=0.1896 loss_action=0.0319 loss_video=0.1577 lr=9.51e-05 speed=0.02 step/s, 2.94 samples/s eta=15:56:23 |
| [2026-07-19 11:30:46,944][fastwam.trainer][INFO] - [train] epoch=0 step=310/1620 loss=0.1948 loss_action=0.0226 loss_video=0.1722 lr=9.47e-05 speed=0.02 step/s, 2.94 samples/s eta=15:49:30 |
| [2026-07-19 11:38:06,180][fastwam.trainer][INFO] - [train] epoch=0 step=320/1620 loss=0.1799 loss_action=0.0225 loss_video=0.1574 lr=9.42e-05 speed=0.02 step/s, 2.94 samples/s eta=15:42:33 |
| [2026-07-19 11:45:20,713][fastwam.trainer][INFO] - [train] epoch=1 step=330/1620 loss=0.1845 loss_action=0.0261 loss_video=0.1583 lr=9.37e-05 speed=0.02 step/s, 2.94 samples/s eta=15:35:16 |
| [2026-07-19 11:52:35,106][fastwam.trainer][INFO] - [train] epoch=1 step=340/1620 loss=0.1752 loss_action=0.0280 loss_video=0.1471 lr=9.32e-05 speed=0.02 step/s, 2.94 samples/s eta=15:27:59 |
| [2026-07-19 11:59:49,005][fastwam.trainer][INFO] - [train] epoch=1 step=350/1620 loss=0.1629 loss_action=0.0193 loss_video=0.1436 lr=9.27e-05 speed=0.02 step/s, 2.94 samples/s eta=15:20:40 |
| [2026-07-19 12:07:02,309][fastwam.trainer][INFO] - [train] epoch=1 step=360/1620 loss=0.1734 loss_action=0.0202 loss_video=0.1533 lr=9.22e-05 speed=0.02 step/s, 2.94 samples/s eta=15:13:19 |
| [2026-07-19 12:14:16,379][fastwam.trainer][INFO] - [train] epoch=1 step=370/1620 loss=0.1813 loss_action=0.0197 loss_video=0.1616 lr=9.16e-05 speed=0.02 step/s, 2.94 samples/s eta=15:06:01 |
| [2026-07-19 12:21:30,925][fastwam.trainer][INFO] - [train] epoch=1 step=380/1620 loss=0.1663 loss_action=0.0239 loss_video=0.1424 lr=9.11e-05 speed=0.02 step/s, 2.94 samples/s eta=14:58:45 |
| [2026-07-19 12:28:45,122][fastwam.trainer][INFO] - [train] epoch=1 step=390/1620 loss=0.1788 loss_action=0.0195 loss_video=0.1593 lr=9.05e-05 speed=0.02 step/s, 2.94 samples/s eta=14:51:28 |
| [2026-07-19 12:35:58,592][fastwam.trainer][INFO] - [train] epoch=1 step=400/1620 loss=0.1597 loss_action=0.0184 loss_video=0.1413 lr=8.99e-05 speed=0.02 step/s, 2.94 samples/s eta=14:44:09 |
| [2026-07-19 12:43:12,859][fastwam.trainer][INFO] - [train] epoch=1 step=410/1620 loss=0.1676 loss_action=0.0246 loss_video=0.1429 lr=8.93e-05 speed=0.02 step/s, 2.94 samples/s eta=14:36:53 |
| [2026-07-19 12:50:26,375][fastwam.trainer][INFO] - [train] epoch=1 step=420/1620 loss=0.1805 loss_action=0.0170 loss_video=0.1635 lr=8.86e-05 speed=0.02 step/s, 2.94 samples/s eta=14:29:34 |
| [2026-07-19 12:57:40,443][fastwam.trainer][INFO] - [train] epoch=1 step=430/1620 loss=0.1765 loss_action=0.0180 loss_video=0.1584 lr=8.80e-05 speed=0.02 step/s, 2.94 samples/s eta=14:22:17 |
| [2026-07-19 13:04:53,349][fastwam.trainer][INFO] - [train] epoch=1 step=440/1620 loss=0.1810 loss_action=0.0190 loss_video=0.1620 lr=8.73e-05 speed=0.02 step/s, 2.94 samples/s eta=14:14:57 |
| [2026-07-19 13:12:07,967][fastwam.trainer][INFO] - [train] epoch=1 step=450/1620 loss=0.1561 loss_action=0.0148 loss_video=0.1413 lr=8.66e-05 speed=0.02 step/s, 2.94 samples/s eta=14:07:42 |
| [2026-07-19 13:19:21,549][fastwam.trainer][INFO] - [train] epoch=1 step=460/1620 loss=0.1645 loss_action=0.0192 loss_video=0.1454 lr=8.59e-05 speed=0.02 step/s, 2.94 samples/s eta=14:00:25 |
| [2026-07-19 13:26:35,288][fastwam.trainer][INFO] - [train] epoch=1 step=470/1620 loss=0.1599 loss_action=0.0149 loss_video=0.1450 lr=8.52e-05 speed=0.02 step/s, 2.94 samples/s eta=13:53:08 |
| [2026-07-19 13:33:47,962][fastwam.trainer][INFO] - [train] epoch=1 step=480/1620 loss=0.1766 loss_action=0.0184 loss_video=0.1581 lr=8.45e-05 speed=0.02 step/s, 2.94 samples/s eta=13:45:48 |
| [2026-07-19 13:41:01,832][fastwam.trainer][INFO] - [train] epoch=1 step=490/1620 loss=0.1813 loss_action=0.0164 loss_video=0.1650 lr=8.37e-05 speed=0.02 step/s, 2.95 samples/s eta=13:38:32 |
| [2026-07-19 13:48:16,471][fastwam.trainer][INFO] - [train] epoch=1 step=500/1620 loss=0.1531 loss_action=0.0179 loss_video=0.1352 lr=8.30e-05 speed=0.02 step/s, 2.95 samples/s eta=13:31:17 |
| [2026-07-19 13:55:29,586][fastwam.trainer][INFO] - [train] epoch=1 step=510/1620 loss=0.1771 loss_action=0.0155 loss_video=0.1616 lr=8.22e-05 speed=0.02 step/s, 2.95 samples/s eta=13:23:59 |
| [2026-07-19 14:02:43,114][fastwam.trainer][INFO] - [train] epoch=1 step=520/1620 loss=0.1809 loss_action=0.0164 loss_video=0.1646 lr=8.14e-05 speed=0.02 step/s, 2.95 samples/s eta=13:16:43 |
| [2026-07-19 14:09:56,212][fastwam.trainer][INFO] - [train] epoch=1 step=530/1620 loss=0.1627 loss_action=0.0136 loss_video=0.1490 lr=8.06e-05 speed=0.02 step/s, 2.95 samples/s eta=13:09:25 |
| [2026-07-19 14:17:06,917][fastwam.trainer][INFO] - [train] epoch=1 step=540/1620 loss=0.1698 loss_action=0.0143 loss_video=0.1555 lr=7.98e-05 speed=0.02 step/s, 2.95 samples/s eta=13:02:03 |
| [2026-07-19 14:24:15,449][fastwam.trainer][INFO] - [train] epoch=1 step=550/1620 loss=0.1709 loss_action=0.0175 loss_video=0.1533 lr=7.90e-05 speed=0.02 step/s, 2.95 samples/s eta=12:54:37 |
| [2026-07-19 14:31:24,487][fastwam.trainer][INFO] - [train] epoch=1 step=560/1620 loss=0.1445 loss_action=0.0146 loss_video=0.1299 lr=7.82e-05 speed=0.02 step/s, 2.95 samples/s eta=12:47:12 |
| [2026-07-19 14:38:40,631][fastwam.trainer][INFO] - [train] epoch=1 step=570/1620 loss=0.1515 loss_action=0.0145 loss_video=0.1370 lr=7.73e-05 speed=0.02 step/s, 2.95 samples/s eta=12:40:01 |
| [2026-07-19 14:45:52,856][fastwam.trainer][INFO] - [train] epoch=1 step=580/1620 loss=0.1431 loss_action=0.0105 loss_video=0.1326 lr=7.65e-05 speed=0.02 step/s, 2.95 samples/s eta=12:32:43 |
| [2026-07-19 14:53:02,161][fastwam.trainer][INFO] - [train] epoch=1 step=590/1620 loss=0.1591 loss_action=0.0134 loss_video=0.1456 lr=7.56e-05 speed=0.02 step/s, 2.95 samples/s eta=12:25:20 |
| [2026-07-19 15:00:12,047][fastwam.trainer][INFO] - [train] epoch=1 step=600/1620 loss=0.1488 loss_action=0.0224 loss_video=0.1265 lr=7.47e-05 speed=0.02 step/s, 2.95 samples/s eta=12:17:59 |
| [2026-07-19 15:07:22,637][fastwam.trainer][INFO] - [train] epoch=1 step=610/1620 loss=0.1678 loss_action=0.0121 loss_video=0.1558 lr=7.38e-05 speed=0.02 step/s, 2.95 samples/s eta=12:10:39 |
| [2026-07-19 15:14:35,001][fastwam.trainer][INFO] - [train] epoch=1 step=620/1620 loss=0.1705 loss_action=0.0148 loss_video=0.1557 lr=7.29e-05 speed=0.02 step/s, 2.95 samples/s eta=12:03:22 |
| [2026-07-19 15:21:42,066][fastwam.trainer][INFO] - [train] epoch=1 step=630/1620 loss=0.1685 loss_action=0.0124 loss_video=0.1561 lr=7.20e-05 speed=0.02 step/s, 2.95 samples/s eta=11:55:57 |
| [2026-07-19 15:28:50,575][fastwam.trainer][INFO] - [train] epoch=1 step=640/1620 loss=0.1534 loss_action=0.0134 loss_video=0.1399 lr=7.11e-05 speed=0.02 step/s, 2.95 samples/s eta=11:48:35 |
| [2026-07-19 15:35:57,137][fastwam.trainer][INFO] - [train] epoch=2 step=650/1620 loss=0.1698 loss_action=0.0165 loss_video=0.1533 lr=7.02e-05 speed=0.02 step/s, 2.95 samples/s eta=11:41:10 |
| [2026-07-19 15:43:04,073][fastwam.trainer][INFO] - [train] epoch=2 step=660/1620 loss=0.1481 loss_action=0.0106 loss_video=0.1374 lr=6.93e-05 speed=0.02 step/s, 2.95 samples/s eta=11:33:47 |
| [2026-07-19 15:50:10,510][fastwam.trainer][INFO] - [train] epoch=2 step=670/1620 loss=0.1602 loss_action=0.0117 loss_video=0.1485 lr=6.83e-05 speed=0.02 step/s, 2.95 samples/s eta=11:26:23 |
| [2026-07-19 15:57:19,187][fastwam.trainer][INFO] - [train] epoch=2 step=680/1620 loss=0.1825 loss_action=0.0120 loss_video=0.1705 lr=6.74e-05 speed=0.02 step/s, 2.95 samples/s eta=11:19:03 |
| [2026-07-19 16:04:24,388][fastwam.trainer][INFO] - [train] epoch=2 step=690/1620 loss=0.1670 loss_action=0.0114 loss_video=0.1556 lr=6.64e-05 speed=0.02 step/s, 2.95 samples/s eta=11:11:38 |
| [2026-07-19 16:11:28,553][fastwam.trainer][INFO] - [train] epoch=2 step=700/1620 loss=0.1326 loss_action=0.0108 loss_video=0.1218 lr=6.55e-05 speed=0.02 step/s, 2.95 samples/s eta=11:04:13 |
| [2026-07-19 16:18:35,751][fastwam.trainer][INFO] - [train] epoch=2 step=710/1620 loss=0.1456 loss_action=0.0112 loss_video=0.1344 lr=6.45e-05 speed=0.02 step/s, 2.96 samples/s eta=10:56:52 |
| [2026-07-19 16:25:43,811][fastwam.trainer][INFO] - [train] epoch=2 step=720/1620 loss=0.1346 loss_action=0.0118 loss_video=0.1228 lr=6.35e-05 speed=0.02 step/s, 2.96 samples/s eta=10:49:33 |
| [2026-07-19 16:32:51,503][fastwam.trainer][INFO] - [train] epoch=2 step=730/1620 loss=0.1555 loss_action=0.0126 loss_video=0.1429 lr=6.26e-05 speed=0.02 step/s, 2.96 samples/s eta=10:42:13 |
| [2026-07-19 16:40:03,823][fastwam.trainer][INFO] - [train] epoch=2 step=740/1620 loss=0.1341 loss_action=0.0116 loss_video=0.1226 lr=6.16e-05 speed=0.02 step/s, 2.96 samples/s eta=10:34:59 |
| [2026-07-19 16:47:13,890][fastwam.trainer][INFO] - [train] epoch=2 step=750/1620 loss=0.1343 loss_action=0.0083 loss_video=0.1259 lr=6.06e-05 speed=0.02 step/s, 2.96 samples/s eta=10:27:43 |
| [2026-07-19 16:54:23,762][fastwam.trainer][INFO] - [train] epoch=2 step=760/1620 loss=0.1518 loss_action=0.0087 loss_video=0.1431 lr=5.96e-05 speed=0.02 step/s, 2.96 samples/s eta=10:20:27 |
| [2026-07-19 17:01:31,803][fastwam.trainer][INFO] - [train] epoch=2 step=770/1620 loss=0.1194 loss_action=0.0098 loss_video=0.1096 lr=5.86e-05 speed=0.02 step/s, 2.96 samples/s eta=10:13:08 |
| [2026-07-19 17:08:42,135][fastwam.trainer][INFO] - [train] epoch=2 step=780/1620 loss=0.1574 loss_action=0.0091 loss_video=0.1483 lr=5.76e-05 speed=0.02 step/s, 2.96 samples/s eta=10:05:53 |
| [2026-07-19 17:15:52,844][fastwam.trainer][INFO] - [train] epoch=2 step=790/1620 loss=0.1290 loss_action=0.0120 loss_video=0.1170 lr=5.66e-05 speed=0.02 step/s, 2.96 samples/s eta=09:58:38 |
| [2026-07-19 17:23:04,407][fastwam.trainer][INFO] - [train] epoch=2 step=800/1620 loss=0.1315 loss_action=0.0090 loss_video=0.1225 lr=5.56e-05 speed=0.02 step/s, 2.96 samples/s eta=09:51:24 |
| [2026-07-19 17:23:20,704][accelerate.accelerator][INFO] - Saving current state to /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800 |
| [2026-07-19 17:23:20,706][accelerate.accelerator][INFO] - Saving DeepSpeed Model and Optimizer |
| [2026-07-19 17:24:44,642][accelerate.accelerator][INFO] - DeepSpeed Model and Optimizer saved to output dir /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800/pytorch_model |
| [2026-07-19 17:24:44,727][accelerate.checkpointing][INFO] - Scheduler state saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800/scheduler.bin |
| [2026-07-19 17:24:44,728][accelerate.checkpointing][INFO] - Sampler state for dataloader 0 saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800/sampler.bin |
| [2026-07-19 17:24:44,847][accelerate.checkpointing][INFO] - Random states saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800/random_states_0.pkl |
| [2026-07-19 17:24:44,888][fastwam.trainer][INFO] - [ckpt] step=800 weights=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/weights/step_000800.pt state=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_000800 |
| [2026-07-19 17:32:03,703][fastwam.trainer][INFO] - [train] epoch=2 step=810/1620 loss=0.1552 loss_action=0.0072 loss_video=0.1480 lr=5.46e-05 speed=0.02 step/s, 2.95 samples/s eta=09:45:58 |
| [2026-07-19 17:39:13,101][fastwam.trainer][INFO] - [train] epoch=2 step=820/1620 loss=0.1517 loss_action=0.0080 loss_video=0.1437 lr=5.36e-05 speed=0.02 step/s, 2.95 samples/s eta=09:38:39 |
| [2026-07-19 17:46:24,220][fastwam.trainer][INFO] - [train] epoch=2 step=830/1620 loss=0.1466 loss_action=0.0076 loss_video=0.1390 lr=5.26e-05 speed=0.02 step/s, 2.95 samples/s eta=09:31:23 |
| [2026-07-19 17:53:34,221][fastwam.trainer][INFO] - [train] epoch=2 step=840/1620 loss=0.1225 loss_action=0.0088 loss_video=0.1137 lr=5.16e-05 speed=0.02 step/s, 2.95 samples/s eta=09:24:05 |
| [2026-07-19 18:00:43,858][fastwam.trainer][INFO] - [train] epoch=2 step=850/1620 loss=0.1345 loss_action=0.0083 loss_video=0.1261 lr=5.06e-05 speed=0.02 step/s, 2.95 samples/s eta=09:16:47 |
| [2026-07-19 18:07:54,096][fastwam.trainer][INFO] - [train] epoch=2 step=860/1620 loss=0.1388 loss_action=0.0074 loss_video=0.1314 lr=4.95e-05 speed=0.02 step/s, 2.95 samples/s eta=09:09:30 |
| [2026-07-19 18:15:03,432][fastwam.trainer][INFO] - [train] epoch=2 step=870/1620 loss=0.1414 loss_action=0.0064 loss_video=0.1350 lr=4.85e-05 speed=0.02 step/s, 2.95 samples/s eta=09:02:12 |
| [2026-07-19 18:22:13,068][fastwam.trainer][INFO] - [train] epoch=2 step=880/1620 loss=0.1209 loss_action=0.0062 loss_video=0.1147 lr=4.75e-05 speed=0.02 step/s, 2.95 samples/s eta=08:54:55 |
| [2026-07-19 18:29:23,885][fastwam.trainer][INFO] - [train] epoch=2 step=890/1620 loss=0.1509 loss_action=0.0058 loss_video=0.1451 lr=4.65e-05 speed=0.02 step/s, 2.95 samples/s eta=08:47:39 |
| [2026-07-19 18:36:29,957][fastwam.trainer][INFO] - [train] epoch=2 step=900/1620 loss=0.1372 loss_action=0.0077 loss_video=0.1295 lr=4.55e-05 speed=0.02 step/s, 2.95 samples/s eta=08:40:19 |
| [2026-07-19 18:43:37,255][fastwam.trainer][INFO] - [train] epoch=2 step=910/1620 loss=0.1392 loss_action=0.0085 loss_video=0.1307 lr=4.45e-05 speed=0.02 step/s, 2.95 samples/s eta=08:33:01 |
| [2026-07-19 18:50:44,550][fastwam.trainer][INFO] - [train] epoch=2 step=920/1620 loss=0.1263 loss_action=0.0063 loss_video=0.1200 lr=4.35e-05 speed=0.02 step/s, 2.95 samples/s eta=08:25:42 |
| [2026-07-19 18:57:58,342][fastwam.trainer][INFO] - [train] epoch=2 step=930/1620 loss=0.1179 loss_action=0.0057 loss_video=0.1122 lr=4.25e-05 speed=0.02 step/s, 2.95 samples/s eta=08:18:29 |
| [2026-07-19 19:05:13,243][fastwam.trainer][INFO] - [train] epoch=2 step=940/1620 loss=0.1259 loss_action=0.0072 loss_video=0.1187 lr=4.15e-05 speed=0.02 step/s, 2.95 samples/s eta=08:11:17 |
| [2026-07-19 19:12:30,414][fastwam.trainer][INFO] - [train] epoch=2 step=950/1620 loss=0.1322 loss_action=0.0048 loss_video=0.1274 lr=4.05e-05 speed=0.02 step/s, 2.95 samples/s eta=08:04:06 |
| [2026-07-19 19:19:44,235][fastwam.trainer][INFO] - [train] epoch=2 step=960/1620 loss=0.1496 loss_action=0.0062 loss_video=0.1433 lr=3.95e-05 speed=0.02 step/s, 2.95 samples/s eta=07:56:53 |
| [2026-07-19 19:26:55,729][fastwam.trainer][INFO] - [train] epoch=2 step=970/1620 loss=0.1385 loss_action=0.0090 loss_video=0.1295 lr=3.85e-05 speed=0.02 step/s, 2.95 samples/s eta=07:49:38 |
| [2026-07-19 19:34:05,800][fastwam.trainer][INFO] - [train] epoch=3 step=980/1620 loss=0.1150 loss_action=0.0056 loss_video=0.1094 lr=3.76e-05 speed=0.02 step/s, 2.95 samples/s eta=07:42:22 |
| [2026-07-19 19:41:18,550][fastwam.trainer][INFO] - [train] epoch=3 step=990/1620 loss=0.1128 loss_action=0.0058 loss_video=0.1070 lr=3.66e-05 speed=0.02 step/s, 2.95 samples/s eta=07:35:08 |
| [2026-07-19 19:48:29,165][fastwam.trainer][INFO] - [train] epoch=3 step=1000/1620 loss=0.1429 loss_action=0.0060 loss_video=0.1369 lr=3.56e-05 speed=0.02 step/s, 2.95 samples/s eta=07:27:53 |
| [2026-07-19 19:55:39,676][fastwam.trainer][INFO] - [train] epoch=3 step=1010/1620 loss=0.1375 loss_action=0.0064 loss_video=0.1311 lr=3.47e-05 speed=0.02 step/s, 2.95 samples/s eta=07:20:37 |
| [2026-07-19 20:02:49,445][fastwam.trainer][INFO] - [train] epoch=3 step=1020/1620 loss=0.1208 loss_action=0.0049 loss_video=0.1159 lr=3.37e-05 speed=0.02 step/s, 2.95 samples/s eta=07:13:22 |
| [2026-07-19 20:10:00,901][fastwam.trainer][INFO] - [train] epoch=3 step=1030/1620 loss=0.1134 loss_action=0.0042 loss_video=0.1091 lr=3.28e-05 speed=0.02 step/s, 2.95 samples/s eta=07:06:07 |
| [2026-07-19 20:17:12,680][fastwam.trainer][INFO] - [train] epoch=3 step=1040/1620 loss=0.1411 loss_action=0.0052 loss_video=0.1359 lr=3.18e-05 speed=0.02 step/s, 2.95 samples/s eta=06:58:53 |
| [2026-07-19 20:24:26,110][fastwam.trainer][INFO] - [train] epoch=3 step=1050/1620 loss=0.1307 loss_action=0.0042 loss_video=0.1265 lr=3.09e-05 speed=0.02 step/s, 2.95 samples/s eta=06:51:40 |
| [2026-07-19 20:31:38,428][fastwam.trainer][INFO] - [train] epoch=3 step=1060/1620 loss=0.1238 loss_action=0.0038 loss_video=0.1200 lr=3.00e-05 speed=0.02 step/s, 2.95 samples/s eta=06:44:26 |
| [2026-07-19 20:38:50,939][fastwam.trainer][INFO] - [train] epoch=3 step=1070/1620 loss=0.1223 loss_action=0.0064 loss_video=0.1159 lr=2.91e-05 speed=0.02 step/s, 2.95 samples/s eta=06:37:12 |
| [2026-07-19 20:46:01,333][fastwam.trainer][INFO] - [train] epoch=3 step=1080/1620 loss=0.1167 loss_action=0.0059 loss_video=0.1108 lr=2.81e-05 speed=0.02 step/s, 2.95 samples/s eta=06:29:58 |
| [2026-07-19 20:53:12,885][fastwam.trainer][INFO] - [train] epoch=3 step=1090/1620 loss=0.1212 loss_action=0.0039 loss_video=0.1173 lr=2.73e-05 speed=0.02 step/s, 2.95 samples/s eta=06:22:43 |
| [2026-07-19 21:00:22,686][fastwam.trainer][INFO] - [train] epoch=3 step=1100/1620 loss=0.1258 loss_action=0.0042 loss_video=0.1216 lr=2.64e-05 speed=0.02 step/s, 2.95 samples/s eta=06:15:28 |
| [2026-07-19 21:07:33,479][fastwam.trainer][INFO] - [train] epoch=3 step=1110/1620 loss=0.1380 loss_action=0.0034 loss_video=0.1346 lr=2.55e-05 speed=0.02 step/s, 2.95 samples/s eta=06:08:14 |
| [2026-07-19 21:14:42,872][fastwam.trainer][INFO] - [train] epoch=3 step=1120/1620 loss=0.1390 loss_action=0.0048 loss_video=0.1341 lr=2.46e-05 speed=0.02 step/s, 2.95 samples/s eta=06:00:59 |
| [2026-07-19 21:21:53,030][fastwam.trainer][INFO] - [train] epoch=3 step=1130/1620 loss=0.1289 loss_action=0.0040 loss_video=0.1249 lr=2.38e-05 speed=0.02 step/s, 2.95 samples/s eta=05:53:45 |
| [2026-07-19 21:29:02,547][fastwam.trainer][INFO] - [train] epoch=3 step=1140/1620 loss=0.1347 loss_action=0.0044 loss_video=0.1303 lr=2.29e-05 speed=0.02 step/s, 2.96 samples/s eta=05:46:30 |
| [2026-07-19 21:36:13,148][fastwam.trainer][INFO] - [train] epoch=3 step=1150/1620 loss=0.1560 loss_action=0.0039 loss_video=0.1521 lr=2.21e-05 speed=0.02 step/s, 2.96 samples/s eta=05:39:16 |
| [2026-07-19 21:43:25,367][fastwam.trainer][INFO] - [train] epoch=3 step=1160/1620 loss=0.1314 loss_action=0.0039 loss_video=0.1275 lr=2.13e-05 speed=0.02 step/s, 2.96 samples/s eta=05:32:02 |
| [2026-07-19 21:50:38,881][fastwam.trainer][INFO] - [train] epoch=3 step=1170/1620 loss=0.1202 loss_action=0.0037 loss_video=0.1164 lr=2.05e-05 speed=0.02 step/s, 2.96 samples/s eta=05:24:49 |
| [2026-07-19 21:57:51,967][fastwam.trainer][INFO] - [train] epoch=3 step=1180/1620 loss=0.1177 loss_action=0.0043 loss_video=0.1134 lr=1.97e-05 speed=0.02 step/s, 2.96 samples/s eta=05:17:36 |
| [2026-07-19 22:05:03,575][fastwam.trainer][INFO] - [train] epoch=3 step=1190/1620 loss=0.1286 loss_action=0.0035 loss_video=0.1251 lr=1.89e-05 speed=0.02 step/s, 2.96 samples/s eta=05:10:23 |
| [2026-07-19 22:12:14,753][fastwam.trainer][INFO] - [train] epoch=3 step=1200/1620 loss=0.1060 loss_action=0.0032 loss_video=0.1028 lr=1.81e-05 speed=0.02 step/s, 2.96 samples/s eta=05:03:09 |
| [2026-07-19 22:19:25,445][fastwam.trainer][INFO] - [train] epoch=3 step=1210/1620 loss=0.1246 loss_action=0.0030 loss_video=0.1216 lr=1.73e-05 speed=0.02 step/s, 2.96 samples/s eta=04:55:55 |
| [2026-07-19 22:26:31,736][fastwam.trainer][INFO] - [train] epoch=3 step=1220/1620 loss=0.1127 loss_action=0.0034 loss_video=0.1093 lr=1.66e-05 speed=0.02 step/s, 2.96 samples/s eta=04:48:40 |
| [2026-07-19 22:33:39,697][fastwam.trainer][INFO] - [train] epoch=3 step=1230/1620 loss=0.1135 loss_action=0.0039 loss_video=0.1096 lr=1.59e-05 speed=0.02 step/s, 2.96 samples/s eta=04:41:25 |
| [2026-07-19 22:40:49,710][fastwam.trainer][INFO] - [train] epoch=3 step=1240/1620 loss=0.1112 loss_action=0.0031 loss_video=0.1081 lr=1.52e-05 speed=0.02 step/s, 2.96 samples/s eta=04:34:11 |
| [2026-07-19 22:47:59,942][fastwam.trainer][INFO] - [train] epoch=3 step=1250/1620 loss=0.1283 loss_action=0.0030 loss_video=0.1253 lr=1.45e-05 speed=0.02 step/s, 2.96 samples/s eta=04:26:57 |
| [2026-07-19 22:55:08,176][fastwam.trainer][INFO] - [train] epoch=3 step=1260/1620 loss=0.1195 loss_action=0.0026 loss_video=0.1169 lr=1.38e-05 speed=0.02 step/s, 2.96 samples/s eta=04:19:43 |
| [2026-07-19 23:02:16,467][fastwam.trainer][INFO] - [train] epoch=3 step=1270/1620 loss=0.1241 loss_action=0.0038 loss_video=0.1203 lr=1.31e-05 speed=0.02 step/s, 2.96 samples/s eta=04:12:29 |
| [2026-07-19 23:09:23,972][fastwam.trainer][INFO] - [train] epoch=3 step=1280/1620 loss=0.1262 loss_action=0.0029 loss_video=0.1233 lr=1.25e-05 speed=0.02 step/s, 2.96 samples/s eta=04:05:15 |
| [2026-07-19 23:16:34,439][fastwam.trainer][INFO] - [train] epoch=3 step=1290/1620 loss=0.1037 loss_action=0.0033 loss_video=0.1005 lr=1.18e-05 speed=0.02 step/s, 2.96 samples/s eta=03:58:01 |
| [2026-07-19 23:23:45,915][fastwam.trainer][INFO] - [train] epoch=4 step=1300/1620 loss=0.1050 loss_action=0.0019 loss_video=0.1031 lr=1.12e-05 speed=0.02 step/s, 2.96 samples/s eta=03:50:48 |
| [2026-07-19 23:30:57,337][fastwam.trainer][INFO] - [train] epoch=4 step=1310/1620 loss=0.1296 loss_action=0.0019 loss_video=0.1277 lr=1.06e-05 speed=0.02 step/s, 2.96 samples/s eta=03:43:35 |
| [2026-07-19 23:38:07,941][fastwam.trainer][INFO] - [train] epoch=4 step=1320/1620 loss=0.1110 loss_action=0.0025 loss_video=0.1085 lr=1.00e-05 speed=0.02 step/s, 2.96 samples/s eta=03:36:22 |
| [2026-07-19 23:45:18,742][fastwam.trainer][INFO] - [train] epoch=4 step=1330/1620 loss=0.1201 loss_action=0.0026 loss_video=0.1175 lr=9.42e-06 speed=0.02 step/s, 2.96 samples/s eta=03:29:09 |
| [2026-07-19 23:52:29,803][fastwam.trainer][INFO] - [train] epoch=4 step=1340/1620 loss=0.1088 loss_action=0.0029 loss_video=0.1059 lr=8.87e-06 speed=0.02 step/s, 2.96 samples/s eta=03:21:56 |
| [2026-07-19 23:59:40,830][fastwam.trainer][INFO] - [train] epoch=4 step=1350/1620 loss=0.1120 loss_action=0.0029 loss_video=0.1091 lr=8.33e-06 speed=0.02 step/s, 2.96 samples/s eta=03:14:43 |
| [2026-07-20 00:06:51,427][fastwam.trainer][INFO] - [train] epoch=4 step=1360/1620 loss=0.1288 loss_action=0.0027 loss_video=0.1262 lr=7.81e-06 speed=0.02 step/s, 2.96 samples/s eta=03:07:29 |
| [2026-07-20 00:14:02,732][fastwam.trainer][INFO] - [train] epoch=4 step=1370/1620 loss=0.1071 loss_action=0.0025 loss_video=0.1046 lr=7.31e-06 speed=0.02 step/s, 2.96 samples/s eta=03:00:17 |
| [2026-07-20 00:21:15,192][fastwam.trainer][INFO] - [train] epoch=4 step=1380/1620 loss=0.1090 loss_action=0.0015 loss_video=0.1076 lr=6.82e-06 speed=0.02 step/s, 2.96 samples/s eta=02:53:04 |
| [2026-07-20 00:28:26,251][fastwam.trainer][INFO] - [train] epoch=4 step=1390/1620 loss=0.1131 loss_action=0.0022 loss_video=0.1109 lr=6.36e-06 speed=0.02 step/s, 2.96 samples/s eta=02:45:51 |
| [2026-07-20 00:35:35,775][fastwam.trainer][INFO] - [train] epoch=4 step=1400/1620 loss=0.1036 loss_action=0.0023 loss_video=0.1013 lr=5.91e-06 speed=0.02 step/s, 2.96 samples/s eta=02:38:38 |
| [2026-07-20 00:42:48,560][fastwam.trainer][INFO] - [train] epoch=4 step=1410/1620 loss=0.1200 loss_action=0.0015 loss_video=0.1185 lr=5.48e-06 speed=0.02 step/s, 2.96 samples/s eta=02:31:25 |
| [2026-07-20 00:50:00,565][fastwam.trainer][INFO] - [train] epoch=4 step=1420/1620 loss=0.1326 loss_action=0.0025 loss_video=0.1301 lr=5.07e-06 speed=0.02 step/s, 2.96 samples/s eta=02:24:12 |
| [2026-07-20 00:57:11,882][fastwam.trainer][INFO] - [train] epoch=4 step=1430/1620 loss=0.1181 loss_action=0.0017 loss_video=0.1164 lr=4.68e-06 speed=0.02 step/s, 2.96 samples/s eta=02:17:00 |
| [2026-07-20 01:04:22,572][fastwam.trainer][INFO] - [train] epoch=4 step=1440/1620 loss=0.0956 loss_action=0.0020 loss_video=0.0936 lr=4.30e-06 speed=0.02 step/s, 2.96 samples/s eta=02:09:47 |
| [2026-07-20 01:11:33,190][fastwam.trainer][INFO] - [train] epoch=4 step=1450/1620 loss=0.1153 loss_action=0.0022 loss_video=0.1131 lr=3.95e-06 speed=0.02 step/s, 2.96 samples/s eta=02:02:34 |
| [2026-07-20 01:18:42,623][fastwam.trainer][INFO] - [train] epoch=4 step=1460/1620 loss=0.0950 loss_action=0.0021 loss_video=0.0929 lr=3.62e-06 speed=0.02 step/s, 2.96 samples/s eta=01:55:21 |
| [2026-07-20 01:25:52,412][fastwam.trainer][INFO] - [train] epoch=4 step=1470/1620 loss=0.1154 loss_action=0.0017 loss_video=0.1137 lr=3.30e-06 speed=0.02 step/s, 2.96 samples/s eta=01:48:08 |
| [2026-07-20 01:33:04,449][fastwam.trainer][INFO] - [train] epoch=4 step=1480/1620 loss=0.1260 loss_action=0.0025 loss_video=0.1235 lr=3.01e-06 speed=0.02 step/s, 2.96 samples/s eta=01:40:55 |
| [2026-07-20 01:40:16,946][fastwam.trainer][INFO] - [train] epoch=4 step=1490/1620 loss=0.1476 loss_action=0.0020 loss_video=0.1456 lr=2.73e-06 speed=0.02 step/s, 2.96 samples/s eta=01:33:43 |
| [2026-07-20 01:47:27,055][fastwam.trainer][INFO] - [train] epoch=4 step=1500/1620 loss=0.1110 loss_action=0.0019 loss_video=0.1090 lr=2.48e-06 speed=0.02 step/s, 2.96 samples/s eta=01:26:30 |
| [2026-07-20 01:54:35,578][fastwam.trainer][INFO] - [train] epoch=4 step=1510/1620 loss=0.1212 loss_action=0.0021 loss_video=0.1191 lr=2.24e-06 speed=0.02 step/s, 2.96 samples/s eta=01:19:17 |
| [2026-07-20 02:01:44,179][fastwam.trainer][INFO] - [train] epoch=4 step=1520/1620 loss=0.1086 loss_action=0.0023 loss_video=0.1063 lr=2.03e-06 speed=0.02 step/s, 2.96 samples/s eta=01:12:04 |
| [2026-07-20 02:08:52,760][fastwam.trainer][INFO] - [train] epoch=4 step=1530/1620 loss=0.1133 loss_action=0.0025 loss_video=0.1107 lr=1.83e-06 speed=0.02 step/s, 2.96 samples/s eta=01:04:52 |
| [2026-07-20 02:16:00,664][fastwam.trainer][INFO] - [train] epoch=4 step=1540/1620 loss=0.1151 loss_action=0.0025 loss_video=0.1126 lr=1.66e-06 speed=0.02 step/s, 2.96 samples/s eta=00:57:39 |
| [2026-07-20 02:23:12,563][fastwam.trainer][INFO] - [train] epoch=4 step=1550/1620 loss=0.0970 loss_action=0.0018 loss_video=0.0951 lr=1.50e-06 speed=0.02 step/s, 2.96 samples/s eta=00:50:27 |
| [2026-07-20 02:30:27,071][fastwam.trainer][INFO] - [train] epoch=4 step=1560/1620 loss=0.0986 loss_action=0.0015 loss_video=0.0971 lr=1.37e-06 speed=0.02 step/s, 2.96 samples/s eta=00:43:14 |
| [2026-07-20 02:37:40,000][fastwam.trainer][INFO] - [train] epoch=4 step=1570/1620 loss=0.1042 loss_action=0.0024 loss_video=0.1019 lr=1.26e-06 speed=0.02 step/s, 2.96 samples/s eta=00:36:02 |
| [2026-07-20 02:44:51,198][fastwam.trainer][INFO] - [train] epoch=4 step=1580/1620 loss=0.1045 loss_action=0.0016 loss_video=0.1028 lr=1.16e-06 speed=0.02 step/s, 2.96 samples/s eta=00:28:49 |
| [2026-07-20 02:52:02,894][fastwam.trainer][INFO] - [train] epoch=4 step=1590/1620 loss=0.1291 loss_action=0.0014 loss_video=0.1277 lr=1.09e-06 speed=0.02 step/s, 2.96 samples/s eta=00:21:37 |
| [2026-07-20 02:59:13,149][fastwam.trainer][INFO] - [train] epoch=4 step=1600/1620 loss=0.1064 loss_action=0.0024 loss_video=0.1041 lr=1.04e-06 speed=0.02 step/s, 2.96 samples/s eta=00:14:24 |
| [2026-07-20 03:06:23,199][fastwam.trainer][INFO] - [train] epoch=4 step=1610/1620 loss=0.1164 loss_action=0.0017 loss_video=0.1147 lr=1.01e-06 speed=0.02 step/s, 2.96 samples/s eta=00:07:12 |
| [2026-07-20 03:13:03,259][fastwam.trainer][INFO] - [train] epoch=4 step=1620/1620 loss=0.1034 loss_action=0.0022 loss_video=0.1012 lr=1.00e-06 speed=0.02 step/s, 2.96 samples/s eta=00:00:00 |
| [2026-07-20 03:13:19,400][accelerate.accelerator][INFO] - Saving current state to /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620 |
| [2026-07-20 03:13:19,402][accelerate.accelerator][INFO] - Saving DeepSpeed Model and Optimizer |
| [2026-07-20 03:14:43,315][accelerate.accelerator][INFO] - DeepSpeed Model and Optimizer saved to output dir /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620/pytorch_model |
| [2026-07-20 03:14:43,511][accelerate.checkpointing][INFO] - Scheduler state saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620/scheduler.bin |
| [2026-07-20 03:14:43,513][accelerate.checkpointing][INFO] - Sampler state for dataloader 0 saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620/sampler.bin |
| [2026-07-20 03:14:43,564][accelerate.checkpointing][INFO] - Random states saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620/random_states_0.pkl |
| [2026-07-20 03:14:43,607][fastwam.trainer][INFO] - [done] max_steps reached step=1620 weights=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/weights/step_001620.pt state=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-tshirt/fastwam/production-20260718T232050Z/checkpoints/state/step_001620 |
|
|
