| [2026-07-19 07:34:05,178][fastwam.models.wan22.helpers.loader][INFO] - Loading Wan2.2-TI2V-5B components... |
| [2026-07-19 07:43:10,011][fastwam.models.wan22.helpers.loader][INFO] - Skipping pretrained text encoder/tokenizer load (`load_text_encoder=False`); training must provide cached `context/context_mask`. |
| [2026-07-19 07:44:39,667][fastwam.models.wan22.helpers.loader][INFO] - Finished loading Wan2.2-TI2V-5B components in 634.49 seconds. |
| [2026-07-19 07:45:44,134][fastwam.models.wan22.action_dit][INFO] - ActionDiT backbone payload policy: {'skip_prefixes': ['action_encoder.', 'head.'], 'alpha_scaling': True, 'interpolation': 'sequential_1d_linear_align_corners_true'} |
| [2026-07-19 07:45:44,630][fastwam.models.wan22.action_dit][INFO] - Loaded ActionDiT backbone from /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/derived/fastwam/actiondit/sha256-7bd65e1986accaaf2e8dd5e59b5e98ef348d7c2f1d88ab2df010eb954a441445/ActionDiT_linear_interp_Wan22_alphascale_1024hdim.pt (keys=820; random_kept_prefixes=['action_encoder.', 'head.']). |
| [2026-07-19 07:45:44,951][fastwam.models.wan22.mot][INFO] - Initialized MoT with experts: ['video', 'action'], num_layers=30 |
| [2026-07-19 07:45:44,954][fastwam.models.wan22.mot][INFO] - Expert 'video': num_params=5.00 B |
| [2026-07-19 07:45:44,957][fastwam.models.wan22.mot][INFO] - Expert 'action': num_params=1.02 B |
| [2026-07-19 07:45:45,807][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:45,807][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:45,956][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,000][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,025][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,032][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,039][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:46,042][datasets][INFO] - PyTorch version 2.9.0+cu128 available. |
| [2026-07-19 07:45:47,417][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpvt2dp8p4/test.c -o /tmp/tmpvt2dp8p4/test.o |
| [2026-07-19 07:45:47,417][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpwpejk80f/test.c -o /tmp/tmpwpejk80f/test.o |
| [2026-07-19 07:45:47,458][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp5b78nms_/test.c -o /tmp/tmp5b78nms_/test.o |
| [2026-07-19 07:45:47,460][root][INFO] - /usr/bin/gcc-11 /tmp/tmpvt2dp8p4/test.o -laio -o /tmp/tmpvt2dp8p4/a.out |
| [2026-07-19 07:45:47,460][root][INFO] - /usr/bin/gcc-11 /tmp/tmpwpejk80f/test.o -laio -o /tmp/tmpwpejk80f/a.out |
| [2026-07-19 07:45:47,464][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpkoo2mfe1/test.c -o /tmp/tmpkoo2mfe1/test.o |
| [2026-07-19 07:45:47,494][root][INFO] - /usr/bin/gcc-11 /tmp/tmp5b78nms_/test.o -laio -o /tmp/tmp5b78nms_/a.out |
| [2026-07-19 07:45:47,559][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpnb48d2tn/test.c -o /tmp/tmpnb48d2tn/test.o |
| [2026-07-19 07:45:47,559][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp5ul2x4fu/test.c -o /tmp/tmp5ul2x4fu/test.o |
| [2026-07-19 07:45:47,569][root][INFO] - /usr/bin/gcc-11 /tmp/tmpkoo2mfe1/test.o -laio -o /tmp/tmpkoo2mfe1/a.out |
| [2026-07-19 07:45:47,571][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpvd429no7/test.c -o /tmp/tmpvd429no7/test.o |
| [2026-07-19 07:45:47,571][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpfvwcgh05/test.c -o /tmp/tmpfvwcgh05/test.o |
| [2026-07-19 07:45:47,575][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpttcgtb_j/test.c -o /tmp/tmpttcgtb_j/test.o |
| [2026-07-19 07:45:47,575][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpsr7dzsek/test.c -o /tmp/tmpsr7dzsek/test.o |
| [2026-07-19 07:45:47,585][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmppzuvnaw8/test.c -o /tmp/tmppzuvnaw8/test.o |
| [2026-07-19 07:45:47,594][root][INFO] - /usr/bin/gcc-11 /tmp/tmpnb48d2tn/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpnb48d2tn/a.out |
| [2026-07-19 07:45:47,594][root][INFO] - /usr/bin/gcc-11 /tmp/tmp5ul2x4fu/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp5ul2x4fu/a.out |
| [2026-07-19 07:45:47,619][root][INFO] - /usr/bin/gcc-11 /tmp/tmpttcgtb_j/test.o -laio -o /tmp/tmpttcgtb_j/a.out |
| [2026-07-19 07:45:47,620][root][INFO] - /usr/bin/gcc-11 /tmp/tmpvd429no7/test.o -laio -o /tmp/tmpvd429no7/a.out |
| [2026-07-19 07:45:47,620][root][INFO] - /usr/bin/gcc-11 /tmp/tmpfvwcgh05/test.o -laio -o /tmp/tmpfvwcgh05/a.out |
| [2026-07-19 07:45:47,620][root][INFO] - /usr/bin/gcc-11 /tmp/tmpsr7dzsek/test.o -laio -o /tmp/tmpsr7dzsek/a.out |
| [2026-07-19 07:45:47,628][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpm_1dtsz8/test.c -o /tmp/tmpm_1dtsz8/test.o |
| [2026-07-19 07:45:47,637][root][INFO] - /usr/bin/gcc-11 /tmp/tmppzuvnaw8/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmppzuvnaw8/a.out |
| [2026-07-19 07:45:47,663][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp7k2fonci/test.c -o /tmp/tmp7k2fonci/test.o |
| [2026-07-19 07:45:47,663][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpadzbd5x6/test.c -o /tmp/tmpadzbd5x6/test.o |
| [2026-07-19 07:45:47,673][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmplxq68jkp/test.c -o /tmp/tmplxq68jkp/test.o |
| [2026-07-19 07:45:47,673][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp9dp_qbka/test.c -o /tmp/tmp9dp_qbka/test.o |
| [2026-07-19 07:45:47,673][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp7za9esmw/test.c -o /tmp/tmp7za9esmw/test.o |
| [2026-07-19 07:45:47,673][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp31u5lur9/test.c -o /tmp/tmp31u5lur9/test.o |
| [2026-07-19 07:45:47,682][root][INFO] - /usr/bin/gcc-11 /tmp/tmpm_1dtsz8/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpm_1dtsz8/a.out |
| [2026-07-19 07:45:47,705][root][INFO] - /usr/bin/gcc-11 /tmp/tmp7k2fonci/test.o -laio -o /tmp/tmp7k2fonci/a.out |
| [2026-07-19 07:45:47,705][root][INFO] - /usr/bin/gcc-11 /tmp/tmpadzbd5x6/test.o -laio -o /tmp/tmpadzbd5x6/a.out |
| [2026-07-19 07:45:47,706][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpmhqnwwym/test.c -o /tmp/tmpmhqnwwym/test.o |
| [2026-07-19 07:45:47,728][root][INFO] - /usr/bin/gcc-11 /tmp/tmp31u5lur9/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp31u5lur9/a.out |
| [2026-07-19 07:45:47,728][root][INFO] - /usr/bin/gcc-11 /tmp/tmplxq68jkp/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmplxq68jkp/a.out |
| [2026-07-19 07:45:47,730][root][INFO] - /usr/bin/gcc-11 /tmp/tmp7za9esmw/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp7za9esmw/a.out |
| [2026-07-19 07:45:47,730][root][INFO] - /usr/bin/gcc-11 /tmp/tmp9dp_qbka/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmp9dp_qbka/a.out |
| [2026-07-19 07:45:47,751][root][INFO] - /usr/bin/gcc-11 /tmp/tmpmhqnwwym/test.o -laio -o /tmp/tmpmhqnwwym/a.out |
| [2026-07-19 07:45:47,753][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp0_eqgijp/test.c -o /tmp/tmp0_eqgijp/test.o |
| [2026-07-19 07:45:47,793][root][INFO] - /usr/bin/gcc-11 /tmp/tmp0_eqgijp/test.o -laio -o /tmp/tmp0_eqgijp/a.out |
| [2026-07-19 07:45:47,798][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpv25l3xio/test.c -o /tmp/tmpv25l3xio/test.o |
| [2026-07-19 07:45:47,798][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp9xet9gs2/test.c -o /tmp/tmp9xet9gs2/test.o |
| [2026-07-19 07:45:47,799][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmpw3crhmns/test.c -o /tmp/tmpw3crhmns/test.o |
| [2026-07-19 07:45:47,799][root][INFO] - /usr/bin/gcc-11 -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -fno-merge-constants -O2 -isystem /opt/synreal/envs/fastwam-py310/include -fPIC -c /tmp/tmp_lcl2zab/test.c -o /tmp/tmp_lcl2zab/test.o |
| [2026-07-19 07:45:47,848][root][INFO] - /usr/bin/gcc-11 /tmp/tmpv25l3xio/test.o -laio -o /tmp/tmpv25l3xio/a.out |
| [2026-07-19 07:45:47,850][root][INFO] - /usr/bin/gcc-11 /tmp/tmpw3crhmns/test.o -laio -o /tmp/tmpw3crhmns/a.out |
| [2026-07-19 07:45:47,850][root][INFO] - /usr/bin/gcc-11 /tmp/tmp9xet9gs2/test.o -laio -o /tmp/tmp9xet9gs2/a.out |
| [2026-07-19 07:45:47,850][root][INFO] - /usr/bin/gcc-11 /tmp/tmp_lcl2zab/test.o -laio -o /tmp/tmp_lcl2zab/a.out |
| [2026-07-19 07:45:57,213][fastwam.datasets.lerobot.robot_video_dataset][INFO] - Calculating dataset stats for normalization... |
| [2026-07-19 07:45:59,576][fastwam.trainer][INFO] - Accelerate training: distributed_type=DistributedType.DEEPSPEED zero_stage=2 world_size=8 process_index=0 cfg_mixed_precision=bf16 accelerator_mixed_precision=bf16 grad_accum=1 grad_clip=1.0000 |
| [2026-07-19 07:45:59,578][fastwam.trainer][INFO] - using accelerator.device=cuda:0 |
| [2026-07-19 07:46:10,321][fastwam.trainer][INFO] - Train/val dataset size: 80432/80432 |
| [2026-07-19 07:46:10,323][fastwam.trainer][INFO] - Setting DiT to train mode and freezing other model components. |
| [2026-07-19 07:46:10,345][fastwam.trainer][INFO] - Starting training with max_steps=3145. |
| [2026-07-19 07:55:59,848][fastwam.trainer][INFO] - [train] epoch=0 step=10/3145 loss=2.0891 loss_action=1.7451 loss_video=0.3440 lr=6.97e-06 speed=0.02 step/s, 2.17 samples/s eta=51:20:08 |
| [2026-07-19 08:05:01,225][fastwam.trainer][INFO] - [train] epoch=0 step=20/3145 loss=1.2676 loss_action=0.9480 loss_video=0.3196 lr=1.33e-05 speed=0.02 step/s, 2.26 samples/s eta=49:04:59 |
| [2026-07-19 08:14:04,366][fastwam.trainer][INFO] - [train] epoch=0 step=30/3145 loss=0.7831 loss_action=0.5400 loss_video=0.2431 lr=1.96e-05 speed=0.02 step/s, 2.29 samples/s eta=48:16:58 |
| [2026-07-19 08:23:06,388][fastwam.trainer][INFO] - [train] epoch=0 step=40/3145 loss=0.5797 loss_action=0.3517 loss_video=0.2279 lr=2.60e-05 speed=0.02 step/s, 2.31 samples/s eta=47:47:00 |
| [2026-07-19 08:32:06,187][fastwam.trainer][INFO] - [train] epoch=0 step=50/3145 loss=0.4917 loss_action=0.2772 loss_video=0.2146 lr=3.23e-05 speed=0.02 step/s, 2.32 samples/s eta=47:23:06 |
| [2026-07-19 08:41:05,595][fastwam.trainer][INFO] - [train] epoch=0 step=60/3145 loss=0.3922 loss_action=0.1973 loss_video=0.1949 lr=3.86e-05 speed=0.02 step/s, 2.33 samples/s eta=47:03:50 |
| [2026-07-19 08:50:06,405][fastwam.trainer][INFO] - [train] epoch=0 step=70/3145 loss=0.3918 loss_action=0.1783 loss_video=0.2135 lr=4.49e-05 speed=0.02 step/s, 2.34 samples/s eta=46:48:32 |
| [2026-07-19 08:59:05,201][fastwam.trainer][INFO] - [train] epoch=0 step=80/3145 loss=0.3523 loss_action=0.1552 loss_video=0.1970 lr=5.13e-05 speed=0.02 step/s, 2.34 samples/s eta=46:33:31 |
| [2026-07-19 09:08:08,173][fastwam.trainer][INFO] - [train] epoch=0 step=90/3145 loss=0.3563 loss_action=0.1572 loss_video=0.1992 lr=5.76e-05 speed=0.02 step/s, 2.34 samples/s eta=46:22:12 |
| [2026-07-19 09:17:07,996][fastwam.trainer][INFO] - [train] epoch=0 step=100/3145 loss=0.2953 loss_action=0.1154 loss_video=0.1798 lr=6.39e-05 speed=0.02 step/s, 2.35 samples/s eta=46:09:45 |
| [2026-07-19 09:26:07,037][fastwam.trainer][INFO] - [train] epoch=0 step=110/3145 loss=0.3120 loss_action=0.1426 loss_video=0.1693 lr=7.03e-05 speed=0.02 step/s, 2.35 samples/s eta=45:57:34 |
| [2026-07-19 09:35:08,412][fastwam.trainer][INFO] - [train] epoch=0 step=120/3145 loss=0.2984 loss_action=0.1216 loss_video=0.1768 lr=7.66e-05 speed=0.02 step/s, 2.35 samples/s eta=45:46:53 |
| [2026-07-19 09:44:08,311][fastwam.trainer][INFO] - [train] epoch=0 step=130/3145 loss=0.3120 loss_action=0.1284 loss_video=0.1835 lr=8.29e-05 speed=0.02 step/s, 2.35 samples/s eta=45:35:54 |
| [2026-07-19 09:53:09,544][fastwam.trainer][INFO] - [train] epoch=0 step=140/3145 loss=0.3027 loss_action=0.1118 loss_video=0.1909 lr=8.92e-05 speed=0.02 step/s, 2.35 samples/s eta=45:25:40 |
| [2026-07-19 10:02:10,736][fastwam.trainer][INFO] - [train] epoch=0 step=150/3145 loss=0.2881 loss_action=0.1013 loss_video=0.1868 lr=9.56e-05 speed=0.02 step/s, 2.35 samples/s eta=45:15:35 |
| [2026-07-19 10:11:11,262][fastwam.trainer][INFO] - [train] epoch=0 step=160/3145 loss=0.2550 loss_action=0.1038 loss_video=0.1511 lr=1.00e-04 speed=0.02 step/s, 2.35 samples/s eta=45:05:26 |
| [2026-07-19 10:20:15,571][fastwam.trainer][INFO] - [train] epoch=0 step=170/3145 loss=0.2737 loss_action=0.1269 loss_video=0.1468 lr=1.00e-04 speed=0.02 step/s, 2.35 samples/s eta=44:56:31 |
| [2026-07-19 10:29:15,293][fastwam.trainer][INFO] - [train] epoch=0 step=180/3145 loss=0.2615 loss_action=0.0792 loss_video=0.1823 lr=1.00e-04 speed=0.02 step/s, 2.35 samples/s eta=44:46:19 |
| [2026-07-19 10:38:15,134][fastwam.trainer][INFO] - [train] epoch=0 step=190/3145 loss=0.2401 loss_action=0.0933 loss_video=0.1468 lr=1.00e-04 speed=0.02 step/s, 2.36 samples/s eta=44:36:17 |
| [2026-07-19 10:47:17,147][fastwam.trainer][INFO] - [train] epoch=0 step=200/3145 loss=0.2425 loss_action=0.0779 loss_video=0.1646 lr=9.99e-05 speed=0.02 step/s, 2.36 samples/s eta=44:26:53 |
| [2026-07-19 10:56:19,560][fastwam.trainer][INFO] - [train] epoch=0 step=210/3145 loss=0.2507 loss_action=0.0720 loss_video=0.1786 lr=9.99e-05 speed=0.02 step/s, 2.36 samples/s eta=44:17:37 |
| [2026-07-19 11:05:19,503][fastwam.trainer][INFO] - [train] epoch=0 step=220/3145 loss=0.2181 loss_action=0.0778 loss_video=0.1403 lr=9.99e-05 speed=0.02 step/s, 2.36 samples/s eta=44:07:49 |
| [2026-07-19 11:14:21,696][fastwam.trainer][INFO] - [train] epoch=0 step=230/3145 loss=0.2581 loss_action=0.0758 loss_video=0.1823 lr=9.99e-05 speed=0.02 step/s, 2.36 samples/s eta=43:58:34 |
| [2026-07-19 11:23:21,641][fastwam.trainer][INFO] - [train] epoch=0 step=240/3145 loss=0.2255 loss_action=0.0656 loss_video=0.1599 lr=9.98e-05 speed=0.02 step/s, 2.36 samples/s eta=43:48:52 |
| [2026-07-19 11:32:18,489][fastwam.trainer][INFO] - [train] epoch=0 step=250/3145 loss=0.2390 loss_action=0.0640 loss_video=0.1750 lr=9.98e-05 speed=0.02 step/s, 2.36 samples/s eta=43:38:39 |
| [2026-07-19 11:41:15,513][fastwam.trainer][INFO] - [train] epoch=0 step=260/3145 loss=0.2440 loss_action=0.0669 loss_video=0.1770 lr=9.97e-05 speed=0.02 step/s, 2.36 samples/s eta=43:28:33 |
| [2026-07-19 11:50:09,720][fastwam.trainer][INFO] - [train] epoch=0 step=270/3145 loss=0.2115 loss_action=0.0610 loss_video=0.1506 lr=9.97e-05 speed=0.02 step/s, 2.36 samples/s eta=43:18:02 |
| [2026-07-19 11:59:06,897][fastwam.trainer][INFO] - [train] epoch=0 step=280/3145 loss=0.2463 loss_action=0.0707 loss_video=0.1755 lr=9.96e-05 speed=0.02 step/s, 2.36 samples/s eta=43:08:08 |
| [2026-07-19 12:08:04,655][fastwam.trainer][INFO] - [train] epoch=0 step=290/3145 loss=0.2216 loss_action=0.0601 loss_video=0.1614 lr=9.95e-05 speed=0.02 step/s, 2.36 samples/s eta=42:58:24 |
| [2026-07-19 12:17:03,397][fastwam.trainer][INFO] - [train] epoch=0 step=300/3145 loss=0.1906 loss_action=0.0556 loss_video=0.1351 lr=9.94e-05 speed=0.02 step/s, 2.36 samples/s eta=42:48:53 |
| [2026-07-19 12:26:03,222][fastwam.trainer][INFO] - [train] epoch=0 step=310/3145 loss=0.2190 loss_action=0.0501 loss_video=0.1689 lr=9.94e-05 speed=0.02 step/s, 2.36 samples/s eta=42:39:33 |
| [2026-07-19 12:35:04,201][fastwam.trainer][INFO] - [train] epoch=0 step=320/3145 loss=0.1994 loss_action=0.0488 loss_video=0.1506 lr=9.93e-05 speed=0.02 step/s, 2.36 samples/s eta=42:30:25 |
| [2026-07-19 12:44:03,741][fastwam.trainer][INFO] - [train] epoch=0 step=330/3145 loss=0.1823 loss_action=0.0373 loss_video=0.1449 lr=9.92e-05 speed=0.02 step/s, 2.36 samples/s eta=42:21:05 |
| [2026-07-19 12:53:04,394][fastwam.trainer][INFO] - [train] epoch=0 step=340/3145 loss=0.2126 loss_action=0.0639 loss_video=0.1487 lr=9.91e-05 speed=0.02 step/s, 2.36 samples/s eta=42:11:55 |
| [2026-07-19 13:02:01,923][fastwam.trainer][INFO] - [train] epoch=0 step=350/3145 loss=0.2031 loss_action=0.0656 loss_video=0.1374 lr=9.90e-05 speed=0.02 step/s, 2.36 samples/s eta=42:02:21 |
| [2026-07-19 13:11:01,776][fastwam.trainer][INFO] - [train] epoch=0 step=360/3145 loss=0.1945 loss_action=0.0517 loss_video=0.1427 lr=9.89e-05 speed=0.02 step/s, 2.36 samples/s eta=41:53:07 |
| [2026-07-19 13:20:00,700][fastwam.trainer][INFO] - [train] epoch=0 step=370/3145 loss=0.1814 loss_action=0.0373 loss_video=0.1441 lr=9.88e-05 speed=0.02 step/s, 2.36 samples/s eta=41:43:47 |
| [2026-07-19 13:29:00,245][fastwam.trainer][INFO] - [train] epoch=0 step=380/3145 loss=0.1979 loss_action=0.0514 loss_video=0.1465 lr=9.86e-05 speed=0.02 step/s, 2.36 samples/s eta=41:34:33 |
| [2026-07-19 13:37:58,674][fastwam.trainer][INFO] - [train] epoch=0 step=390/3145 loss=0.2054 loss_action=0.0493 loss_video=0.1562 lr=9.85e-05 speed=0.02 step/s, 2.36 samples/s eta=41:25:11 |
| [2026-07-19 13:46:55,386][fastwam.trainer][INFO] - [train] epoch=0 step=400/3145 loss=0.1745 loss_action=0.0433 loss_video=0.1312 lr=9.84e-05 speed=0.02 step/s, 2.37 samples/s eta=41:15:39 |
| [2026-07-19 13:55:54,936][fastwam.trainer][INFO] - [train] epoch=0 step=410/3145 loss=0.1862 loss_action=0.0447 loss_video=0.1415 lr=9.83e-05 speed=0.02 step/s, 2.37 samples/s eta=41:06:27 |
| [2026-07-19 14:04:51,667][fastwam.trainer][INFO] - [train] epoch=0 step=420/3145 loss=0.1914 loss_action=0.0394 loss_video=0.1520 lr=9.81e-05 speed=0.02 step/s, 2.37 samples/s eta=40:56:58 |
| [2026-07-19 14:13:42,064][fastwam.trainer][INFO] - [train] epoch=0 step=430/3145 loss=0.1886 loss_action=0.0416 loss_video=0.1470 lr=9.80e-05 speed=0.02 step/s, 2.37 samples/s eta=40:46:50 |
| [2026-07-19 14:22:28,801][fastwam.trainer][INFO] - [train] epoch=0 step=440/3145 loss=0.2016 loss_action=0.0465 loss_video=0.1550 lr=9.78e-05 speed=0.02 step/s, 2.37 samples/s eta=40:36:23 |
| [2026-07-19 14:31:16,763][fastwam.trainer][INFO] - [train] epoch=0 step=450/3145 loss=0.1759 loss_action=0.0423 loss_video=0.1335 lr=9.77e-05 speed=0.02 step/s, 2.37 samples/s eta=40:26:08 |
| [2026-07-19 14:40:04,110][fastwam.trainer][INFO] - [train] epoch=0 step=460/3145 loss=0.1743 loss_action=0.0471 loss_video=0.1272 lr=9.75e-05 speed=0.02 step/s, 2.37 samples/s eta=40:15:53 |
| [2026-07-19 14:48:52,069][fastwam.trainer][INFO] - [train] epoch=0 step=470/3145 loss=0.1667 loss_action=0.0319 loss_video=0.1348 lr=9.73e-05 speed=0.02 step/s, 2.37 samples/s eta=40:05:45 |
| [2026-07-19 14:57:39,255][fastwam.trainer][INFO] - [train] epoch=0 step=480/3145 loss=0.1944 loss_action=0.0393 loss_video=0.1551 lr=9.72e-05 speed=0.02 step/s, 2.37 samples/s eta=39:55:37 |
| [2026-07-19 15:06:25,781][fastwam.trainer][INFO] - [train] epoch=0 step=490/3145 loss=0.1961 loss_action=0.0469 loss_video=0.1492 lr=9.70e-05 speed=0.02 step/s, 2.37 samples/s eta=39:45:28 |
| [2026-07-19 15:15:12,864][fastwam.trainer][INFO] - [train] epoch=0 step=500/3145 loss=0.1813 loss_action=0.0497 loss_video=0.1315 lr=9.68e-05 speed=0.02 step/s, 2.38 samples/s eta=39:35:25 |
| [2026-07-19 15:24:01,838][fastwam.trainer][INFO] - [train] epoch=0 step=510/3145 loss=0.1851 loss_action=0.0323 loss_video=0.1528 lr=9.66e-05 speed=0.02 step/s, 2.38 samples/s eta=39:25:36 |
| [2026-07-19 15:32:47,598][fastwam.trainer][INFO] - [train] epoch=0 step=520/3145 loss=0.1890 loss_action=0.0328 loss_video=0.1562 lr=9.64e-05 speed=0.02 step/s, 2.38 samples/s eta=39:15:32 |
| [2026-07-19 15:41:35,385][fastwam.trainer][INFO] - [train] epoch=0 step=530/3145 loss=0.1819 loss_action=0.0342 loss_video=0.1477 lr=9.62e-05 speed=0.02 step/s, 2.38 samples/s eta=39:05:41 |
| [2026-07-19 15:50:23,469][fastwam.trainer][INFO] - [train] epoch=0 step=540/3145 loss=0.1872 loss_action=0.0334 loss_video=0.1538 lr=9.60e-05 speed=0.02 step/s, 2.38 samples/s eta=38:55:54 |
| [2026-07-19 15:59:10,827][fastwam.trainer][INFO] - [train] epoch=0 step=550/3145 loss=0.1751 loss_action=0.0398 loss_video=0.1353 lr=9.58e-05 speed=0.02 step/s, 2.38 samples/s eta=38:46:06 |
| [2026-07-19 16:07:57,562][fastwam.trainer][INFO] - [train] epoch=0 step=560/3145 loss=0.1523 loss_action=0.0327 loss_video=0.1196 lr=9.56e-05 speed=0.02 step/s, 2.38 samples/s eta=38:36:17 |
| [2026-07-19 16:16:43,008][fastwam.trainer][INFO] - [train] epoch=0 step=570/3145 loss=0.1733 loss_action=0.0347 loss_video=0.1386 lr=9.54e-05 speed=0.02 step/s, 2.38 samples/s eta=38:26:24 |
| [2026-07-19 16:25:28,879][fastwam.trainer][INFO] - [train] epoch=0 step=580/3145 loss=0.1539 loss_action=0.0333 loss_video=0.1206 lr=9.52e-05 speed=0.02 step/s, 2.38 samples/s eta=38:16:35 |
| [2026-07-19 16:34:14,179][fastwam.trainer][INFO] - [train] epoch=0 step=590/3145 loss=0.1692 loss_action=0.0283 loss_video=0.1408 lr=9.50e-05 speed=0.02 step/s, 2.38 samples/s eta=38:06:47 |
| [2026-07-19 16:43:00,604][fastwam.trainer][INFO] - [train] epoch=0 step=600/3145 loss=0.1603 loss_action=0.0337 loss_video=0.1266 lr=9.47e-05 speed=0.02 step/s, 2.38 samples/s eta=37:57:05 |
| [2026-07-19 16:51:47,405][fastwam.trainer][INFO] - [train] epoch=0 step=610/3145 loss=0.1850 loss_action=0.0432 loss_video=0.1418 lr=9.45e-05 speed=0.02 step/s, 2.39 samples/s eta=37:47:26 |
| [2026-07-19 17:00:34,105][fastwam.trainer][INFO] - [train] epoch=0 step=620/3145 loss=0.1837 loss_action=0.0370 loss_video=0.1467 lr=9.42e-05 speed=0.02 step/s, 2.39 samples/s eta=37:37:49 |
| [2026-07-19 17:09:20,290][fastwam.trainer][INFO] - [train] epoch=1 step=630/3145 loss=0.1795 loss_action=0.0348 loss_video=0.1447 lr=9.40e-05 speed=0.02 step/s, 2.39 samples/s eta=37:28:11 |
| [2026-07-19 17:18:07,740][fastwam.trainer][INFO] - [train] epoch=1 step=640/3145 loss=0.1690 loss_action=0.0402 loss_video=0.1289 lr=9.38e-05 speed=0.02 step/s, 2.39 samples/s eta=37:18:40 |
| [2026-07-19 17:26:54,085][fastwam.trainer][INFO] - [train] epoch=1 step=650/3145 loss=0.1841 loss_action=0.0293 loss_video=0.1548 lr=9.35e-05 speed=0.02 step/s, 2.39 samples/s eta=37:09:06 |
| [2026-07-19 17:35:38,735][fastwam.trainer][INFO] - [train] epoch=1 step=660/3145 loss=0.1617 loss_action=0.0279 loss_video=0.1338 lr=9.32e-05 speed=0.02 step/s, 2.39 samples/s eta=36:59:27 |
| [2026-07-19 17:44:27,653][fastwam.trainer][INFO] - [train] epoch=1 step=670/3145 loss=0.1671 loss_action=0.0250 loss_video=0.1421 lr=9.30e-05 speed=0.02 step/s, 2.39 samples/s eta=36:50:05 |
| [2026-07-19 17:53:14,272][fastwam.trainer][INFO] - [train] epoch=1 step=680/3145 loss=0.1989 loss_action=0.0282 loss_video=0.1706 lr=9.27e-05 speed=0.02 step/s, 2.39 samples/s eta=36:40:36 |
| [2026-07-19 18:02:02,554][fastwam.trainer][INFO] - [train] epoch=1 step=690/3145 loss=0.1802 loss_action=0.0287 loss_video=0.1515 lr=9.24e-05 speed=0.02 step/s, 2.39 samples/s eta=36:31:14 |
| [2026-07-19 18:10:49,632][fastwam.trainer][INFO] - [train] epoch=1 step=700/3145 loss=0.1425 loss_action=0.0249 loss_video=0.1176 lr=9.21e-05 speed=0.02 step/s, 2.39 samples/s eta=36:21:49 |
| [2026-07-19 18:19:37,474][fastwam.trainer][INFO] - [train] epoch=1 step=710/3145 loss=0.1596 loss_action=0.0273 loss_video=0.1323 lr=9.19e-05 speed=0.02 step/s, 2.39 samples/s eta=36:12:28 |
| [2026-07-19 18:28:24,304][fastwam.trainer][INFO] - [train] epoch=1 step=720/3145 loss=0.1509 loss_action=0.0272 loss_video=0.1237 lr=9.16e-05 speed=0.02 step/s, 2.39 samples/s eta=36:03:04 |
| [2026-07-19 18:37:11,256][fastwam.trainer][INFO] - [train] epoch=1 step=730/3145 loss=0.1852 loss_action=0.0311 loss_video=0.1541 lr=9.13e-05 speed=0.02 step/s, 2.39 samples/s eta=35:53:42 |
| [2026-07-19 18:45:59,362][fastwam.trainer][INFO] - [train] epoch=1 step=740/3145 loss=0.1535 loss_action=0.0291 loss_video=0.1244 lr=9.10e-05 speed=0.02 step/s, 2.39 samples/s eta=35:44:24 |
| [2026-07-19 18:54:45,697][fastwam.trainer][INFO] - [train] epoch=1 step=750/3145 loss=0.1492 loss_action=0.0203 loss_video=0.1290 lr=9.07e-05 speed=0.02 step/s, 2.39 samples/s eta=35:35:01 |
| [2026-07-19 19:03:35,206][fastwam.trainer][INFO] - [train] epoch=1 step=760/3145 loss=0.1612 loss_action=0.0210 loss_video=0.1402 lr=9.04e-05 speed=0.02 step/s, 2.39 samples/s eta=35:25:49 |
| [2026-07-19 19:12:20,756][fastwam.trainer][INFO] - [train] epoch=1 step=770/3145 loss=0.1382 loss_action=0.0266 loss_video=0.1116 lr=9.01e-05 speed=0.02 step/s, 2.39 samples/s eta=35:16:26 |
| [2026-07-19 19:21:06,287][fastwam.trainer][INFO] - [train] epoch=1 step=780/3145 loss=0.1764 loss_action=0.0283 loss_video=0.1481 lr=8.98e-05 speed=0.02 step/s, 2.39 samples/s eta=35:07:04 |
| [2026-07-19 19:29:54,211][fastwam.trainer][INFO] - [train] epoch=1 step=790/3145 loss=0.1470 loss_action=0.0253 loss_video=0.1217 lr=8.94e-05 speed=0.02 step/s, 2.39 samples/s eta=34:57:49 |
| [2026-07-19 19:38:42,253][fastwam.trainer][INFO] - [train] epoch=1 step=800/3145 loss=0.1523 loss_action=0.0293 loss_video=0.1230 lr=8.91e-05 speed=0.02 step/s, 2.40 samples/s eta=34:48:36 |
| [2026-07-19 19:47:32,092][fastwam.trainer][INFO] - [train] epoch=1 step=810/3145 loss=0.1592 loss_action=0.0259 loss_video=0.1333 lr=8.88e-05 speed=0.02 step/s, 2.40 samples/s eta=34:39:28 |
| [2026-07-19 19:56:18,522][fastwam.trainer][INFO] - [train] epoch=1 step=820/3145 loss=0.1584 loss_action=0.0224 loss_video=0.1360 lr=8.85e-05 speed=0.02 step/s, 2.40 samples/s eta=34:30:12 |
| [2026-07-19 20:05:05,692][fastwam.trainer][INFO] - [train] epoch=1 step=830/3145 loss=0.1578 loss_action=0.0244 loss_video=0.1335 lr=8.81e-05 speed=0.02 step/s, 2.40 samples/s eta=34:20:58 |
| [2026-07-19 20:13:52,463][fastwam.trainer][INFO] - [train] epoch=1 step=840/3145 loss=0.1396 loss_action=0.0287 loss_video=0.1109 lr=8.78e-05 speed=0.02 step/s, 2.40 samples/s eta=34:11:43 |
| [2026-07-19 20:22:39,021][fastwam.trainer][INFO] - [train] epoch=1 step=850/3145 loss=0.1497 loss_action=0.0253 loss_video=0.1244 lr=8.74e-05 speed=0.02 step/s, 2.40 samples/s eta=34:02:29 |
| [2026-07-19 20:31:27,017][fastwam.trainer][INFO] - [train] epoch=1 step=860/3145 loss=0.1514 loss_action=0.0270 loss_video=0.1244 lr=8.71e-05 speed=0.02 step/s, 2.40 samples/s eta=33:53:19 |
| [2026-07-19 20:40:15,116][fastwam.trainer][INFO] - [train] epoch=1 step=870/3145 loss=0.1688 loss_action=0.0265 loss_video=0.1424 lr=8.67e-05 speed=0.02 step/s, 2.40 samples/s eta=33:44:10 |
| [2026-07-19 20:49:02,425][fastwam.trainer][INFO] - [train] epoch=1 step=880/3145 loss=0.1321 loss_action=0.0242 loss_video=0.1079 lr=8.64e-05 speed=0.02 step/s, 2.40 samples/s eta=33:34:59 |
| [2026-07-19 20:57:50,080][fastwam.trainer][INFO] - [train] epoch=1 step=890/3145 loss=0.1688 loss_action=0.0307 loss_video=0.1381 lr=8.60e-05 speed=0.02 step/s, 2.40 samples/s eta=33:25:50 |
| [2026-07-19 21:06:37,159][fastwam.trainer][INFO] - [train] epoch=1 step=900/3145 loss=0.1509 loss_action=0.0248 loss_video=0.1261 lr=8.56e-05 speed=0.02 step/s, 2.40 samples/s eta=33:16:40 |
| [2026-07-19 21:15:25,830][fastwam.trainer][INFO] - [train] epoch=1 step=910/3145 loss=0.1477 loss_action=0.0219 loss_video=0.1257 lr=8.53e-05 speed=0.02 step/s, 2.40 samples/s eta=33:07:34 |
| [2026-07-19 21:24:13,496][fastwam.trainer][INFO] - [train] epoch=1 step=920/3145 loss=0.1477 loss_action=0.0203 loss_video=0.1273 lr=8.49e-05 speed=0.02 step/s, 2.40 samples/s eta=32:58:26 |
| [2026-07-19 21:33:03,135][fastwam.trainer][INFO] - [train] epoch=1 step=930/3145 loss=0.1323 loss_action=0.0232 loss_video=0.1091 lr=8.45e-05 speed=0.02 step/s, 2.40 samples/s eta=32:49:23 |
| [2026-07-19 21:41:53,047][fastwam.trainer][INFO] - [train] epoch=1 step=940/3145 loss=0.1482 loss_action=0.0237 loss_video=0.1245 lr=8.42e-05 speed=0.02 step/s, 2.40 samples/s eta=32:40:21 |
| [2026-07-19 21:50:37,989][fastwam.trainer][INFO] - [train] epoch=1 step=950/3145 loss=0.1371 loss_action=0.0138 loss_video=0.1233 lr=8.38e-05 speed=0.02 step/s, 2.40 samples/s eta=32:31:08 |
| [2026-07-19 21:59:23,763][fastwam.trainer][INFO] - [train] epoch=1 step=960/3145 loss=0.1634 loss_action=0.0278 loss_video=0.1356 lr=8.34e-05 speed=0.02 step/s, 2.40 samples/s eta=32:21:58 |
| [2026-07-19 22:08:08,903][fastwam.trainer][INFO] - [train] epoch=1 step=970/3145 loss=0.1583 loss_action=0.0241 loss_video=0.1342 lr=8.30e-05 speed=0.02 step/s, 2.40 samples/s eta=32:12:46 |
| [2026-07-19 22:16:57,065][fastwam.trainer][INFO] - [train] epoch=1 step=980/3145 loss=0.1418 loss_action=0.0313 loss_video=0.1105 lr=8.26e-05 speed=0.02 step/s, 2.40 samples/s eta=32:03:42 |
| [2026-07-19 22:25:42,301][fastwam.trainer][INFO] - [train] epoch=1 step=990/3145 loss=0.1316 loss_action=0.0221 loss_video=0.1094 lr=8.22e-05 speed=0.02 step/s, 2.40 samples/s eta=31:54:32 |
| [2026-07-19 22:34:26,733][fastwam.trainer][INFO] - [train] epoch=1 step=1000/3145 loss=0.1507 loss_action=0.0210 loss_video=0.1296 lr=8.18e-05 speed=0.02 step/s, 2.40 samples/s eta=31:45:20 |
| [2026-07-19 22:43:13,110][fastwam.trainer][INFO] - [train] epoch=1 step=1010/3145 loss=0.1582 loss_action=0.0277 loss_video=0.1305 lr=8.14e-05 speed=0.02 step/s, 2.40 samples/s eta=31:36:13 |
| [2026-07-19 22:52:01,429][fastwam.trainer][INFO] - [train] epoch=1 step=1020/3145 loss=0.1379 loss_action=0.0201 loss_video=0.1178 lr=8.10e-05 speed=0.02 step/s, 2.40 samples/s eta=31:27:11 |
| [2026-07-19 23:00:46,674][fastwam.trainer][INFO] - [train] epoch=1 step=1030/3145 loss=0.1338 loss_action=0.0169 loss_video=0.1168 lr=8.06e-05 speed=0.02 step/s, 2.40 samples/s eta=31:18:02 |
| [2026-07-19 23:09:32,540][fastwam.trainer][INFO] - [train] epoch=1 step=1040/3145 loss=0.1526 loss_action=0.0208 loss_video=0.1317 lr=8.02e-05 speed=0.02 step/s, 2.40 samples/s eta=31:08:56 |
| [2026-07-19 23:18:18,319][fastwam.trainer][INFO] - [train] epoch=1 step=1050/3145 loss=0.1453 loss_action=0.0240 loss_video=0.1213 lr=7.97e-05 speed=0.02 step/s, 2.40 samples/s eta=30:59:49 |
| [2026-07-19 23:27:01,790][fastwam.trainer][INFO] - [train] epoch=1 step=1060/3145 loss=0.1392 loss_action=0.0152 loss_video=0.1241 lr=7.93e-05 speed=0.02 step/s, 2.40 samples/s eta=30:50:38 |
| [2026-07-19 23:35:47,425][fastwam.trainer][INFO] - [train] epoch=1 step=1070/3145 loss=0.1319 loss_action=0.0217 loss_video=0.1102 lr=7.89e-05 speed=0.02 step/s, 2.40 samples/s eta=30:41:32 |
| [2026-07-19 23:44:33,914][fastwam.trainer][INFO] - [train] epoch=1 step=1080/3145 loss=0.1257 loss_action=0.0189 loss_video=0.1068 lr=7.85e-05 speed=0.02 step/s, 2.40 samples/s eta=30:32:28 |
| [2026-07-19 23:53:21,097][fastwam.trainer][INFO] - [train] epoch=1 step=1090/3145 loss=0.1371 loss_action=0.0186 loss_video=0.1185 lr=7.80e-05 speed=0.02 step/s, 2.40 samples/s eta=30:23:26 |
| [2026-07-20 00:02:07,327][fastwam.trainer][INFO] - [train] epoch=1 step=1100/3145 loss=0.1371 loss_action=0.0201 loss_video=0.1169 lr=7.76e-05 speed=0.02 step/s, 2.40 samples/s eta=30:14:22 |
| [2026-07-20 00:10:52,854][fastwam.trainer][INFO] - [train] epoch=1 step=1110/3145 loss=0.1480 loss_action=0.0220 loss_video=0.1260 lr=7.72e-05 speed=0.02 step/s, 2.40 samples/s eta=30:05:17 |
| [2026-07-20 00:19:39,223][fastwam.trainer][INFO] - [train] epoch=1 step=1120/3145 loss=0.1560 loss_action=0.0200 loss_video=0.1359 lr=7.67e-05 speed=0.02 step/s, 2.41 samples/s eta=29:56:14 |
| [2026-07-20 00:28:23,006][fastwam.trainer][INFO] - [train] epoch=1 step=1130/3145 loss=0.1323 loss_action=0.0147 loss_video=0.1175 lr=7.63e-05 speed=0.02 step/s, 2.41 samples/s eta=29:47:07 |
| [2026-07-20 00:37:08,833][fastwam.trainer][INFO] - [train] epoch=1 step=1140/3145 loss=0.1555 loss_action=0.0225 loss_video=0.1331 lr=7.58e-05 speed=0.02 step/s, 2.41 samples/s eta=29:38:04 |
| [2026-07-20 00:45:55,598][fastwam.trainer][INFO] - [train] epoch=1 step=1150/3145 loss=0.1593 loss_action=0.0182 loss_video=0.1411 lr=7.54e-05 speed=0.02 step/s, 2.41 samples/s eta=29:29:03 |
| [2026-07-20 00:54:45,135][fastwam.trainer][INFO] - [train] epoch=1 step=1160/3145 loss=0.1494 loss_action=0.0190 loss_video=0.1305 lr=7.49e-05 speed=0.02 step/s, 2.41 samples/s eta=29:20:06 |
| [2026-07-20 01:03:34,262][fastwam.trainer][INFO] - [train] epoch=1 step=1170/3145 loss=0.1438 loss_action=0.0244 loss_video=0.1194 lr=7.45e-05 speed=0.02 step/s, 2.41 samples/s eta=29:11:09 |
| [2026-07-20 01:12:15,696][fastwam.trainer][INFO] - [train] epoch=1 step=1180/3145 loss=0.1472 loss_action=0.0238 loss_video=0.1234 lr=7.40e-05 speed=0.02 step/s, 2.41 samples/s eta=29:02:00 |
| [2026-07-20 01:20:44,419][fastwam.trainer][INFO] - [train] epoch=1 step=1190/3145 loss=0.1474 loss_action=0.0259 loss_video=0.1215 lr=7.36e-05 speed=0.02 step/s, 2.41 samples/s eta=28:52:30 |
| [2026-07-20 01:29:15,464][fastwam.trainer][INFO] - [train] epoch=1 step=1200/3145 loss=0.1332 loss_action=0.0199 loss_video=0.1133 lr=7.31e-05 speed=0.02 step/s, 2.41 samples/s eta=28:43:05 |
| [2026-07-20 01:37:48,551][fastwam.trainer][INFO] - [train] epoch=1 step=1210/3145 loss=0.1469 loss_action=0.0200 loss_video=0.1268 lr=7.26e-05 speed=0.02 step/s, 2.41 samples/s eta=28:33:43 |
| [2026-07-20 01:46:18,788][fastwam.trainer][INFO] - [train] epoch=1 step=1220/3145 loss=0.1402 loss_action=0.0176 loss_video=0.1225 lr=7.22e-05 speed=0.02 step/s, 2.41 samples/s eta=28:24:19 |
| [2026-07-20 01:54:51,037][fastwam.trainer][INFO] - [train] epoch=1 step=1230/3145 loss=0.1293 loss_action=0.0187 loss_video=0.1106 lr=7.17e-05 speed=0.02 step/s, 2.41 samples/s eta=28:14:58 |
| [2026-07-20 02:03:22,997][fastwam.trainer][INFO] - [train] epoch=1 step=1240/3145 loss=0.1259 loss_action=0.0189 loss_video=0.1070 lr=7.12e-05 speed=0.02 step/s, 2.41 samples/s eta=28:05:38 |
| [2026-07-20 02:11:55,728][fastwam.trainer][INFO] - [train] epoch=1 step=1250/3145 loss=0.1458 loss_action=0.0221 loss_video=0.1237 lr=7.08e-05 speed=0.02 step/s, 2.41 samples/s eta=27:56:19 |
| [2026-07-20 02:20:27,894][fastwam.trainer][INFO] - [train] epoch=2 step=1260/3145 loss=0.1360 loss_action=0.0182 loss_video=0.1178 lr=7.03e-05 speed=0.02 step/s, 2.41 samples/s eta=27:47:01 |
| [2026-07-20 02:28:59,051][fastwam.trainer][INFO] - [train] epoch=2 step=1270/3145 loss=0.1478 loss_action=0.0195 loss_video=0.1283 lr=6.98e-05 speed=0.02 step/s, 2.41 samples/s eta=27:37:41 |
| [2026-07-20 02:37:30,180][fastwam.trainer][INFO] - [train] epoch=2 step=1280/3145 loss=0.1375 loss_action=0.0162 loss_video=0.1213 lr=6.93e-05 speed=0.02 step/s, 2.41 samples/s eta=27:28:23 |
| [2026-07-20 02:45:59,752][fastwam.trainer][INFO] - [train] epoch=2 step=1290/3145 loss=0.1145 loss_action=0.0164 loss_video=0.0981 lr=6.88e-05 speed=0.02 step/s, 2.41 samples/s eta=27:19:02 |
| [2026-07-20 02:54:29,701][fastwam.trainer][INFO] - [train] epoch=2 step=1300/3145 loss=0.1237 loss_action=0.0155 loss_video=0.1082 lr=6.84e-05 speed=0.02 step/s, 2.42 samples/s eta=27:09:44 |
| [2026-07-20 03:03:01,815][fastwam.trainer][INFO] - [train] epoch=2 step=1310/3145 loss=0.1505 loss_action=0.0176 loss_video=0.1328 lr=6.79e-05 speed=0.02 step/s, 2.42 samples/s eta=27:00:29 |
| [2026-07-20 03:11:35,131][fastwam.trainer][INFO] - [train] epoch=2 step=1320/3145 loss=0.1274 loss_action=0.0163 loss_video=0.1111 lr=6.74e-05 speed=0.02 step/s, 2.42 samples/s eta=26:51:16 |
| [2026-07-20 03:20:06,111][fastwam.trainer][INFO] - [train] epoch=2 step=1330/3145 loss=0.1443 loss_action=0.0184 loss_video=0.1259 lr=6.69e-05 speed=0.02 step/s, 2.42 samples/s eta=26:42:00 |
| [2026-07-20 03:28:36,147][fastwam.trainer][INFO] - [train] epoch=2 step=1340/3145 loss=0.1350 loss_action=0.0191 loss_video=0.1160 lr=6.64e-05 speed=0.02 step/s, 2.42 samples/s eta=26:32:45 |
| [2026-07-20 03:37:08,081][fastwam.trainer][INFO] - [train] epoch=2 step=1350/3145 loss=0.1193 loss_action=0.0120 loss_video=0.1073 lr=6.59e-05 speed=0.02 step/s, 2.42 samples/s eta=26:23:32 |
| [2026-07-20 03:45:41,886][fastwam.trainer][INFO] - [train] epoch=2 step=1360/3145 loss=0.1408 loss_action=0.0151 loss_video=0.1257 lr=6.54e-05 speed=0.02 step/s, 2.42 samples/s eta=26:14:22 |
| [2026-07-20 03:54:13,915][fastwam.trainer][INFO] - [train] epoch=2 step=1370/3145 loss=0.1301 loss_action=0.0180 loss_video=0.1121 lr=6.49e-05 speed=0.02 step/s, 2.42 samples/s eta=26:05:11 |
| [2026-07-20 04:02:44,491][fastwam.trainer][INFO] - [train] epoch=2 step=1380/3145 loss=0.1287 loss_action=0.0140 loss_video=0.1148 lr=6.44e-05 speed=0.02 step/s, 2.42 samples/s eta=25:55:58 |
| [2026-07-20 04:11:13,941][fastwam.trainer][INFO] - [train] epoch=2 step=1390/3145 loss=0.1198 loss_action=0.0151 loss_video=0.1047 lr=6.39e-05 speed=0.02 step/s, 2.42 samples/s eta=25:46:44 |
| [2026-07-20 04:19:44,379][fastwam.trainer][INFO] - [train] epoch=2 step=1400/3145 loss=0.1158 loss_action=0.0117 loss_video=0.1041 lr=6.34e-05 speed=0.02 step/s, 2.42 samples/s eta=25:37:33 |
| [2026-07-20 04:28:14,075][fastwam.trainer][INFO] - [train] epoch=2 step=1410/3145 loss=0.1356 loss_action=0.0140 loss_video=0.1216 lr=6.29e-05 speed=0.02 step/s, 2.42 samples/s eta=25:28:21 |
| [2026-07-20 04:36:44,678][fastwam.trainer][INFO] - [train] epoch=2 step=1420/3145 loss=0.1379 loss_action=0.0126 loss_video=0.1253 lr=6.24e-05 speed=0.02 step/s, 2.42 samples/s eta=25:19:10 |
| [2026-07-20 04:45:15,973][fastwam.trainer][INFO] - [train] epoch=2 step=1430/3145 loss=0.1381 loss_action=0.0179 loss_video=0.1203 lr=6.19e-05 speed=0.02 step/s, 2.42 samples/s eta=25:10:01 |
| [2026-07-20 04:53:46,130][fastwam.trainer][INFO] - [train] epoch=2 step=1440/3145 loss=0.1127 loss_action=0.0159 loss_video=0.0967 lr=6.14e-05 speed=0.02 step/s, 2.42 samples/s eta=25:00:52 |
| [2026-07-20 05:02:16,776][fastwam.trainer][INFO] - [train] epoch=2 step=1450/3145 loss=0.1249 loss_action=0.0149 loss_video=0.1100 lr=6.09e-05 speed=0.02 step/s, 2.42 samples/s eta=24:51:43 |
| [2026-07-20 05:10:49,853][fastwam.trainer][INFO] - [train] epoch=2 step=1460/3145 loss=0.1134 loss_action=0.0134 loss_video=0.1000 lr=6.04e-05 speed=0.02 step/s, 2.42 samples/s eta=24:42:38 |
| [2026-07-20 05:19:21,703][fastwam.trainer][INFO] - [train] epoch=2 step=1470/3145 loss=0.1281 loss_action=0.0120 loss_video=0.1161 lr=5.99e-05 speed=0.02 step/s, 2.43 samples/s eta=24:33:31 |
| [2026-07-20 05:27:52,581][fastwam.trainer][INFO] - [train] epoch=2 step=1480/3145 loss=0.1305 loss_action=0.0166 loss_video=0.1138 lr=5.94e-05 speed=0.02 step/s, 2.43 samples/s eta=24:24:25 |
| [2026-07-20 05:36:25,531][fastwam.trainer][INFO] - [train] epoch=2 step=1490/3145 loss=0.1589 loss_action=0.0193 loss_video=0.1396 lr=5.88e-05 speed=0.02 step/s, 2.43 samples/s eta=24:15:20 |
| [2026-07-20 05:44:55,930][fastwam.trainer][INFO] - [train] epoch=2 step=1500/3145 loss=0.1184 loss_action=0.0143 loss_video=0.1041 lr=5.83e-05 speed=0.02 step/s, 2.43 samples/s eta=24:06:14 |
| [2026-07-20 05:45:12,537][accelerate.accelerator][INFO] - Saving current state to /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500 |
| [2026-07-20 05:45:12,539][accelerate.accelerator][INFO] - Saving DeepSpeed Model and Optimizer |
| [2026-07-20 05:46:37,236][accelerate.accelerator][INFO] - DeepSpeed Model and Optimizer saved to output dir /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500/pytorch_model |
| [2026-07-20 05:46:37,283][accelerate.checkpointing][INFO] - Scheduler state saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500/scheduler.bin |
| [2026-07-20 05:46:37,284][accelerate.checkpointing][INFO] - Sampler state for dataloader 0 saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500/sampler.bin |
| [2026-07-20 05:46:37,457][accelerate.checkpointing][INFO] - Random states saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500/random_states_0.pkl |
| [2026-07-20 05:46:37,504][fastwam.trainer][INFO] - [ckpt] step=1500 weights=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/weights/step_001500.pt state=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_001500 |
| [2026-07-20 05:55:07,677][fastwam.trainer][INFO] - [train] epoch=2 step=1510/3145 loss=0.1328 loss_action=0.0197 loss_video=0.1131 lr=5.78e-05 speed=0.02 step/s, 2.42 samples/s eta=23:58:58 |
| [2026-07-20 06:03:40,645][fastwam.trainer][INFO] - [train] epoch=2 step=1520/3145 loss=0.1063 loss_action=0.0126 loss_video=0.0937 lr=5.73e-05 speed=0.02 step/s, 2.42 samples/s eta=23:49:53 |
| [2026-07-20 06:12:13,813][fastwam.trainer][INFO] - [train] epoch=2 step=1530/3145 loss=0.1275 loss_action=0.0138 loss_video=0.1137 lr=5.68e-05 speed=0.02 step/s, 2.42 samples/s eta=23:40:50 |
| [2026-07-20 06:20:45,521][fastwam.trainer][INFO] - [train] epoch=2 step=1540/3145 loss=0.1283 loss_action=0.0135 loss_video=0.1147 lr=5.63e-05 speed=0.02 step/s, 2.43 samples/s eta=23:31:45 |
| [2026-07-20 06:29:17,316][fastwam.trainer][INFO] - [train] epoch=2 step=1550/3145 loss=0.1100 loss_action=0.0109 loss_video=0.0991 lr=5.57e-05 speed=0.02 step/s, 2.43 samples/s eta=23:22:41 |
| [2026-07-20 06:37:49,087][fastwam.trainer][INFO] - [train] epoch=2 step=1560/3145 loss=0.1238 loss_action=0.0116 loss_video=0.1122 lr=5.52e-05 speed=0.02 step/s, 2.43 samples/s eta=23:13:37 |
| [2026-07-20 06:46:22,940][fastwam.trainer][INFO] - [train] epoch=2 step=1570/3145 loss=0.1235 loss_action=0.0140 loss_video=0.1095 lr=5.47e-05 speed=0.02 step/s, 2.43 samples/s eta=23:04:36 |
| [2026-07-20 06:54:53,688][fastwam.trainer][INFO] - [train] epoch=2 step=1580/3145 loss=0.1232 loss_action=0.0194 loss_video=0.1038 lr=5.42e-05 speed=0.02 step/s, 2.43 samples/s eta=22:55:32 |
| [2026-07-20 07:03:26,182][fastwam.trainer][INFO] - [train] epoch=2 step=1590/3145 loss=0.1325 loss_action=0.0094 loss_video=0.1231 lr=5.37e-05 speed=0.02 step/s, 2.43 samples/s eta=22:46:30 |
| [2026-07-20 07:11:59,693][fastwam.trainer][INFO] - [train] epoch=2 step=1600/3145 loss=0.1171 loss_action=0.0121 loss_video=0.1050 lr=5.32e-05 speed=0.02 step/s, 2.43 samples/s eta=22:37:29 |
| [2026-07-20 07:20:32,785][fastwam.trainer][INFO] - [train] epoch=2 step=1610/3145 loss=0.1195 loss_action=0.0094 loss_video=0.1101 lr=5.26e-05 speed=0.02 step/s, 2.43 samples/s eta=22:28:29 |
| [2026-07-20 07:29:03,896][fastwam.trainer][INFO] - [train] epoch=2 step=1620/3145 loss=0.1151 loss_action=0.0134 loss_video=0.1017 lr=5.21e-05 speed=0.02 step/s, 2.43 samples/s eta=22:19:27 |
| [2026-07-20 07:37:34,983][fastwam.trainer][INFO] - [train] epoch=2 step=1630/3145 loss=0.1094 loss_action=0.0127 loss_video=0.0967 lr=5.16e-05 speed=0.02 step/s, 2.43 samples/s eta=22:10:25 |
| [2026-07-20 07:46:07,007][fastwam.trainer][INFO] - [train] epoch=2 step=1640/3145 loss=0.1297 loss_action=0.0103 loss_video=0.1193 lr=5.11e-05 speed=0.02 step/s, 2.43 samples/s eta=22:01:24 |
| [2026-07-20 07:54:37,724][fastwam.trainer][INFO] - [train] epoch=2 step=1650/3145 loss=0.1169 loss_action=0.0129 loss_video=0.1040 lr=5.06e-05 speed=0.02 step/s, 2.43 samples/s eta=21:52:23 |
| [2026-07-20 08:03:09,489][fastwam.trainer][INFO] - [train] epoch=2 step=1660/3145 loss=0.0950 loss_action=0.0106 loss_video=0.0843 lr=5.00e-05 speed=0.02 step/s, 2.43 samples/s eta=21:43:23 |
| [2026-07-20 08:11:40,958][fastwam.trainer][INFO] - [train] epoch=2 step=1670/3145 loss=0.1405 loss_action=0.0106 loss_video=0.1299 lr=4.95e-05 speed=0.02 step/s, 2.43 samples/s eta=21:34:23 |
| [2026-07-20 08:20:13,736][fastwam.trainer][INFO] - [train] epoch=2 step=1680/3145 loss=0.1090 loss_action=0.0113 loss_video=0.0977 lr=4.90e-05 speed=0.02 step/s, 2.43 samples/s eta=21:25:24 |
| [2026-07-20 08:28:45,124][fastwam.trainer][INFO] - [train] epoch=2 step=1690/3145 loss=0.1157 loss_action=0.0077 loss_video=0.1080 lr=4.85e-05 speed=0.02 step/s, 2.43 samples/s eta=21:16:25 |
| [2026-07-20 08:37:18,284][fastwam.trainer][INFO] - [train] epoch=2 step=1700/3145 loss=0.1441 loss_action=0.0181 loss_video=0.1261 lr=4.80e-05 speed=0.02 step/s, 2.43 samples/s eta=21:07:27 |
| [2026-07-20 08:45:48,688][fastwam.trainer][INFO] - [train] epoch=2 step=1710/3145 loss=0.1118 loss_action=0.0103 loss_video=0.1015 lr=4.74e-05 speed=0.02 step/s, 2.43 samples/s eta=20:58:28 |
| [2026-07-20 08:54:18,490][fastwam.trainer][INFO] - [train] epoch=2 step=1720/3145 loss=0.1132 loss_action=0.0125 loss_video=0.1007 lr=4.69e-05 speed=0.02 step/s, 2.43 samples/s eta=20:49:28 |
| [2026-07-20 09:02:49,492][fastwam.trainer][INFO] - [train] epoch=2 step=1730/3145 loss=0.1143 loss_action=0.0123 loss_video=0.1020 lr=4.64e-05 speed=0.02 step/s, 2.43 samples/s eta=20:40:29 |
| [2026-07-20 09:11:19,784][fastwam.trainer][INFO] - [train] epoch=2 step=1740/3145 loss=0.1134 loss_action=0.0103 loss_video=0.1031 lr=4.59e-05 speed=0.02 step/s, 2.43 samples/s eta=20:31:31 |
| [2026-07-20 09:19:53,921][fastwam.trainer][INFO] - [train] epoch=2 step=1750/3145 loss=0.1229 loss_action=0.0133 loss_video=0.1095 lr=4.54e-05 speed=0.02 step/s, 2.43 samples/s eta=20:22:35 |
| [2026-07-20 09:28:24,872][fastwam.trainer][INFO] - [train] epoch=2 step=1760/3145 loss=0.1171 loss_action=0.0115 loss_video=0.1056 lr=4.48e-05 speed=0.02 step/s, 2.43 samples/s eta=20:13:38 |
| [2026-07-20 09:36:55,460][fastwam.trainer][INFO] - [train] epoch=2 step=1770/3145 loss=0.1301 loss_action=0.0112 loss_video=0.1189 lr=4.43e-05 speed=0.02 step/s, 2.43 samples/s eta=20:04:40 |
| [2026-07-20 09:45:25,332][fastwam.trainer][INFO] - [train] epoch=2 step=1780/3145 loss=0.1161 loss_action=0.0139 loss_video=0.1022 lr=4.38e-05 speed=0.02 step/s, 2.44 samples/s eta=19:55:43 |
| [2026-07-20 09:53:53,973][fastwam.trainer][INFO] - [train] epoch=2 step=1790/3145 loss=0.1120 loss_action=0.0122 loss_video=0.0997 lr=4.33e-05 speed=0.02 step/s, 2.44 samples/s eta=19:46:44 |
| [2026-07-20 10:02:24,086][fastwam.trainer][INFO] - [train] epoch=2 step=1800/3145 loss=0.1231 loss_action=0.0093 loss_video=0.1138 lr=4.28e-05 speed=0.02 step/s, 2.44 samples/s eta=19:37:47 |
| [2026-07-20 10:10:54,620][fastwam.trainer][INFO] - [train] epoch=2 step=1810/3145 loss=0.1346 loss_action=0.0077 loss_video=0.1269 lr=4.23e-05 speed=0.02 step/s, 2.44 samples/s eta=19:28:51 |
| [2026-07-20 10:19:23,465][fastwam.trainer][INFO] - [train] epoch=2 step=1820/3145 loss=0.1008 loss_action=0.0088 loss_video=0.0921 lr=4.18e-05 speed=0.02 step/s, 2.44 samples/s eta=19:19:53 |
| [2026-07-20 10:27:53,809][fastwam.trainer][INFO] - [train] epoch=2 step=1830/3145 loss=0.1039 loss_action=0.0066 loss_video=0.0973 lr=4.12e-05 speed=0.02 step/s, 2.44 samples/s eta=19:10:57 |
| [2026-07-20 10:36:24,552][fastwam.trainer][INFO] - [train] epoch=2 step=1840/3145 loss=0.1272 loss_action=0.0088 loss_video=0.1184 lr=4.07e-05 speed=0.02 step/s, 2.44 samples/s eta=19:02:02 |
| [2026-07-20 10:44:54,414][fastwam.trainer][INFO] - [train] epoch=2 step=1850/3145 loss=0.1103 loss_action=0.0108 loss_video=0.0996 lr=4.02e-05 speed=0.02 step/s, 2.44 samples/s eta=18:53:06 |
| [2026-07-20 10:53:24,907][fastwam.trainer][INFO] - [train] epoch=2 step=1860/3145 loss=0.1148 loss_action=0.0100 loss_video=0.1048 lr=3.97e-05 speed=0.02 step/s, 2.44 samples/s eta=18:44:11 |
| [2026-07-20 11:01:54,428][fastwam.trainer][INFO] - [train] epoch=2 step=1870/3145 loss=0.1150 loss_action=0.0088 loss_video=0.1062 lr=3.92e-05 speed=0.02 step/s, 2.44 samples/s eta=18:35:16 |
| [2026-07-20 11:10:25,286][fastwam.trainer][INFO] - [train] epoch=2 step=1880/3145 loss=0.1104 loss_action=0.0089 loss_video=0.1015 lr=3.87e-05 speed=0.02 step/s, 2.44 samples/s eta=18:26:22 |
| [2026-07-20 11:18:55,253][fastwam.trainer][INFO] - [train] epoch=3 step=1890/3145 loss=0.1137 loss_action=0.0093 loss_video=0.1044 lr=3.82e-05 speed=0.02 step/s, 2.44 samples/s eta=18:17:27 |
| [2026-07-20 11:27:25,455][fastwam.trainer][INFO] - [train] epoch=3 step=1900/3145 loss=0.1258 loss_action=0.0105 loss_video=0.1153 lr=3.77e-05 speed=0.02 step/s, 2.44 samples/s eta=18:08:33 |
| [2026-07-20 11:35:53,554][fastwam.trainer][INFO] - [train] epoch=3 step=1910/3145 loss=0.1171 loss_action=0.0094 loss_video=0.1077 lr=3.72e-05 speed=0.02 step/s, 2.44 samples/s eta=17:59:38 |
| [2026-07-20 11:44:20,529][fastwam.trainer][INFO] - [train] epoch=3 step=1920/3145 loss=0.1178 loss_action=0.0065 loss_video=0.1113 lr=3.67e-05 speed=0.02 step/s, 2.44 samples/s eta=17:50:42 |
| [2026-07-20 11:52:50,637][fastwam.trainer][INFO] - [train] epoch=3 step=1930/3145 loss=0.1028 loss_action=0.0095 loss_video=0.0933 lr=3.62e-05 speed=0.02 step/s, 2.44 samples/s eta=17:41:48 |
| [2026-07-20 12:01:21,624][fastwam.trainer][INFO] - [train] epoch=3 step=1940/3145 loss=0.1293 loss_action=0.0079 loss_video=0.1214 lr=3.57e-05 speed=0.02 step/s, 2.44 samples/s eta=17:32:56 |
| [2026-07-20 12:09:53,403][fastwam.trainer][INFO] - [train] epoch=3 step=1950/3145 loss=0.1147 loss_action=0.0100 loss_video=0.1046 lr=3.52e-05 speed=0.02 step/s, 2.44 samples/s eta=17:24:04 |
| [2026-07-20 12:18:21,725][fastwam.trainer][INFO] - [train] epoch=3 step=1960/3145 loss=0.1209 loss_action=0.0101 loss_video=0.1108 lr=3.47e-05 speed=0.02 step/s, 2.44 samples/s eta=17:15:10 |
| [2026-07-20 12:26:51,743][fastwam.trainer][INFO] - [train] epoch=3 step=1970/3145 loss=0.1210 loss_action=0.0084 loss_video=0.1125 lr=3.42e-05 speed=0.02 step/s, 2.44 samples/s eta=17:06:17 |
| [2026-07-20 12:35:22,814][fastwam.trainer][INFO] - [train] epoch=3 step=1980/3145 loss=0.1149 loss_action=0.0085 loss_video=0.1064 lr=3.37e-05 speed=0.02 step/s, 2.44 samples/s eta=16:57:26 |
| [2026-07-20 12:43:53,254][fastwam.trainer][INFO] - [train] epoch=3 step=1990/3145 loss=0.0917 loss_action=0.0101 loss_video=0.0816 lr=3.32e-05 speed=0.02 step/s, 2.44 samples/s eta=16:48:34 |
| [2026-07-20 12:52:25,642][fastwam.trainer][INFO] - [train] epoch=3 step=2000/3145 loss=0.1140 loss_action=0.0080 loss_video=0.1060 lr=3.27e-05 speed=0.02 step/s, 2.44 samples/s eta=16:39:43 |
| [2026-07-20 13:00:55,705][fastwam.trainer][INFO] - [train] epoch=3 step=2010/3145 loss=0.1123 loss_action=0.0070 loss_video=0.1053 lr=3.23e-05 speed=0.02 step/s, 2.44 samples/s eta=16:30:52 |
| [2026-07-20 13:09:26,457][fastwam.trainer][INFO] - [train] epoch=3 step=2020/3145 loss=0.0940 loss_action=0.0064 loss_video=0.0876 lr=3.18e-05 speed=0.02 step/s, 2.44 samples/s eta=16:22:01 |
| [2026-07-20 13:17:55,134][fastwam.trainer][INFO] - [train] epoch=3 step=2030/3145 loss=0.1217 loss_action=0.0058 loss_video=0.1159 lr=3.13e-05 speed=0.02 step/s, 2.44 samples/s eta=16:13:09 |
| [2026-07-20 13:26:25,430][fastwam.trainer][INFO] - [train] epoch=3 step=2040/3145 loss=0.1080 loss_action=0.0077 loss_video=0.1002 lr=3.08e-05 speed=0.02 step/s, 2.44 samples/s eta=16:04:18 |
| [2026-07-20 13:34:57,205][fastwam.trainer][INFO] - [train] epoch=3 step=2050/3145 loss=0.1063 loss_action=0.0073 loss_video=0.0990 lr=3.03e-05 speed=0.02 step/s, 2.44 samples/s eta=15:55:28 |
| [2026-07-20 13:43:26,498][fastwam.trainer][INFO] - [train] epoch=3 step=2060/3145 loss=0.1012 loss_action=0.0086 loss_video=0.0926 lr=2.99e-05 speed=0.02 step/s, 2.45 samples/s eta=15:46:37 |
| [2026-07-20 13:51:57,911][fastwam.trainer][INFO] - [train] epoch=3 step=2070/3145 loss=0.1173 loss_action=0.0067 loss_video=0.1106 lr=2.94e-05 speed=0.02 step/s, 2.45 samples/s eta=15:37:47 |
| [2026-07-20 14:00:31,167][fastwam.trainer][INFO] - [train] epoch=3 step=2080/3145 loss=0.1009 loss_action=0.0066 loss_video=0.0943 lr=2.89e-05 speed=0.02 step/s, 2.45 samples/s eta=15:28:58 |
| [2026-07-20 14:09:02,980][fastwam.trainer][INFO] - [train] epoch=3 step=2090/3145 loss=0.1083 loss_action=0.0071 loss_video=0.1012 lr=2.85e-05 speed=0.02 step/s, 2.45 samples/s eta=15:20:09 |
| [2026-07-20 14:17:34,111][fastwam.trainer][INFO] - [train] epoch=3 step=2100/3145 loss=0.1182 loss_action=0.0065 loss_video=0.1117 lr=2.80e-05 speed=0.02 step/s, 2.45 samples/s eta=15:11:20 |
| [2026-07-20 14:26:05,386][fastwam.trainer][INFO] - [train] epoch=3 step=2110/3145 loss=0.0947 loss_action=0.0064 loss_video=0.0883 lr=2.75e-05 speed=0.02 step/s, 2.45 samples/s eta=15:02:31 |
| [2026-07-20 14:34:33,128][fastwam.trainer][INFO] - [train] epoch=3 step=2120/3145 loss=0.1075 loss_action=0.0097 loss_video=0.0978 lr=2.71e-05 speed=0.02 step/s, 2.45 samples/s eta=14:53:40 |
| [2026-07-20 14:43:02,467][fastwam.trainer][INFO] - [train] epoch=3 step=2130/3145 loss=0.0905 loss_action=0.0059 loss_video=0.0845 lr=2.66e-05 speed=0.02 step/s, 2.45 samples/s eta=14:44:50 |
| [2026-07-20 14:51:29,959][fastwam.trainer][INFO] - [train] epoch=3 step=2140/3145 loss=0.1140 loss_action=0.0089 loss_video=0.1051 lr=2.62e-05 speed=0.02 step/s, 2.45 samples/s eta=14:36:00 |
| [2026-07-20 15:00:01,090][fastwam.trainer][INFO] - [train] epoch=3 step=2150/3145 loss=0.1008 loss_action=0.0063 loss_video=0.0945 lr=2.57e-05 speed=0.02 step/s, 2.45 samples/s eta=14:27:11 |
| [2026-07-20 15:08:32,110][fastwam.trainer][INFO] - [train] epoch=3 step=2160/3145 loss=0.0994 loss_action=0.0052 loss_video=0.0942 lr=2.53e-05 speed=0.02 step/s, 2.45 samples/s eta=14:18:23 |
| [2026-07-20 15:17:02,178][fastwam.trainer][INFO] - [train] epoch=3 step=2170/3145 loss=0.1019 loss_action=0.0061 loss_video=0.0958 lr=2.48e-05 speed=0.02 step/s, 2.45 samples/s eta=14:09:34 |
| [2026-07-20 15:25:33,741][fastwam.trainer][INFO] - [train] epoch=3 step=2180/3145 loss=0.0990 loss_action=0.0057 loss_video=0.0934 lr=2.44e-05 speed=0.02 step/s, 2.45 samples/s eta=14:00:47 |
| [2026-07-20 15:34:06,242][fastwam.trainer][INFO] - [train] epoch=3 step=2190/3145 loss=0.0992 loss_action=0.0060 loss_video=0.0933 lr=2.39e-05 speed=0.02 step/s, 2.45 samples/s eta=13:51:59 |
| [2026-07-20 15:42:35,357][fastwam.trainer][INFO] - [train] epoch=3 step=2200/3145 loss=0.0986 loss_action=0.0049 loss_video=0.0938 lr=2.35e-05 speed=0.02 step/s, 2.45 samples/s eta=13:43:11 |
| [2026-07-20 15:51:04,513][fastwam.trainer][INFO] - [train] epoch=3 step=2210/3145 loss=0.0973 loss_action=0.0081 loss_video=0.0892 lr=2.31e-05 speed=0.02 step/s, 2.45 samples/s eta=13:34:22 |
| [2026-07-20 15:59:35,162][fastwam.trainer][INFO] - [train] epoch=3 step=2220/3145 loss=0.1173 loss_action=0.0062 loss_video=0.1112 lr=2.26e-05 speed=0.02 step/s, 2.45 samples/s eta=13:25:35 |
| [2026-07-20 16:08:08,716][fastwam.trainer][INFO] - [train] epoch=3 step=2230/3145 loss=0.1135 loss_action=0.0059 loss_video=0.1076 lr=2.22e-05 speed=0.02 step/s, 2.45 samples/s eta=13:16:49 |
| [2026-07-20 16:16:40,062][fastwam.trainer][INFO] - [train] epoch=3 step=2240/3145 loss=0.0950 loss_action=0.0053 loss_video=0.0897 lr=2.18e-05 speed=0.02 step/s, 2.45 samples/s eta=13:08:02 |
| [2026-07-20 16:25:08,602][fastwam.trainer][INFO] - [train] epoch=3 step=2250/3145 loss=0.1153 loss_action=0.0054 loss_video=0.1098 lr=2.13e-05 speed=0.02 step/s, 2.45 samples/s eta=12:59:14 |
| [2026-07-20 16:33:35,442][fastwam.trainer][INFO] - [train] epoch=3 step=2260/3145 loss=0.0990 loss_action=0.0051 loss_video=0.0939 lr=2.09e-05 speed=0.02 step/s, 2.45 samples/s eta=12:50:25 |
| [2026-07-20 16:42:02,982][fastwam.trainer][INFO] - [train] epoch=3 step=2270/3145 loss=0.0939 loss_action=0.0064 loss_video=0.0875 lr=2.05e-05 speed=0.02 step/s, 2.45 samples/s eta=12:41:37 |
| [2026-07-20 16:50:35,532][fastwam.trainer][INFO] - [train] epoch=3 step=2280/3145 loss=0.0882 loss_action=0.0056 loss_video=0.0826 lr=2.01e-05 speed=0.02 step/s, 2.45 samples/s eta=12:32:51 |
| [2026-07-20 16:59:06,554][fastwam.trainer][INFO] - [train] epoch=3 step=2290/3145 loss=0.1092 loss_action=0.0066 loss_video=0.1026 lr=1.97e-05 speed=0.02 step/s, 2.45 samples/s eta=12:24:05 |
| [2026-07-20 17:07:36,291][fastwam.trainer][INFO] - [train] epoch=3 step=2300/3145 loss=0.1066 loss_action=0.0055 loss_video=0.1011 lr=1.93e-05 speed=0.02 step/s, 2.45 samples/s eta=12:15:18 |
| [2026-07-20 17:16:06,313][fastwam.trainer][INFO] - [train] epoch=3 step=2310/3145 loss=0.1125 loss_action=0.0052 loss_video=0.1073 lr=1.89e-05 speed=0.02 step/s, 2.45 samples/s eta=12:06:32 |
| [2026-07-20 17:24:35,954][fastwam.trainer][INFO] - [train] epoch=3 step=2320/3145 loss=0.0991 loss_action=0.0066 loss_video=0.0926 lr=1.85e-05 speed=0.02 step/s, 2.45 samples/s eta=11:57:45 |
| [2026-07-20 17:33:03,672][fastwam.trainer][INFO] - [train] epoch=3 step=2330/3145 loss=0.1164 loss_action=0.0053 loss_video=0.1111 lr=1.81e-05 speed=0.02 step/s, 2.45 samples/s eta=11:48:58 |
| [2026-07-20 17:41:36,454][fastwam.trainer][INFO] - [train] epoch=3 step=2340/3145 loss=0.0922 loss_action=0.0049 loss_video=0.0873 lr=1.77e-05 speed=0.02 step/s, 2.45 samples/s eta=11:40:13 |
| [2026-07-20 17:50:04,948][fastwam.trainer][INFO] - [train] epoch=3 step=2350/3145 loss=0.0999 loss_action=0.0055 loss_video=0.0944 lr=1.73e-05 speed=0.02 step/s, 2.45 samples/s eta=11:31:27 |
| [2026-07-20 17:58:36,377][fastwam.trainer][INFO] - [train] epoch=3 step=2360/3145 loss=0.1184 loss_action=0.0041 loss_video=0.1144 lr=1.69e-05 speed=0.02 step/s, 2.45 samples/s eta=11:22:41 |
| [2026-07-20 18:07:05,947][fastwam.trainer][INFO] - [train] epoch=3 step=2370/3145 loss=0.1055 loss_action=0.0053 loss_video=0.1002 lr=1.65e-05 speed=0.02 step/s, 2.45 samples/s eta=11:13:55 |
| [2026-07-20 18:15:37,388][fastwam.trainer][INFO] - [train] epoch=3 step=2380/3145 loss=0.1044 loss_action=0.0045 loss_video=0.0999 lr=1.62e-05 speed=0.02 step/s, 2.45 samples/s eta=11:05:10 |
| [2026-07-20 18:24:06,239][fastwam.trainer][INFO] - [train] epoch=3 step=2390/3145 loss=0.1051 loss_action=0.0046 loss_video=0.1005 lr=1.58e-05 speed=0.02 step/s, 2.45 samples/s eta=10:56:25 |
| [2026-07-20 18:32:36,553][fastwam.trainer][INFO] - [train] epoch=3 step=2400/3145 loss=0.1079 loss_action=0.0045 loss_video=0.1033 lr=1.54e-05 speed=0.02 step/s, 2.45 samples/s eta=10:47:39 |
| [2026-07-20 18:41:10,721][fastwam.trainer][INFO] - [train] epoch=3 step=2410/3145 loss=0.1018 loss_action=0.0065 loss_video=0.0953 lr=1.51e-05 speed=0.02 step/s, 2.45 samples/s eta=10:38:56 |
| [2026-07-20 18:49:39,433][fastwam.trainer][INFO] - [train] epoch=3 step=2420/3145 loss=0.0910 loss_action=0.0045 loss_video=0.0865 lr=1.47e-05 speed=0.02 step/s, 2.45 samples/s eta=10:30:10 |
| [2026-07-20 18:58:11,018][fastwam.trainer][INFO] - [train] epoch=3 step=2430/3145 loss=0.0969 loss_action=0.0049 loss_video=0.0920 lr=1.43e-05 speed=0.02 step/s, 2.45 samples/s eta=10:21:26 |
| [2026-07-20 19:06:41,505][fastwam.trainer][INFO] - [train] epoch=3 step=2440/3145 loss=0.1097 loss_action=0.0033 loss_video=0.1064 lr=1.40e-05 speed=0.02 step/s, 2.45 samples/s eta=10:12:41 |
| [2026-07-20 19:15:11,763][fastwam.trainer][INFO] - [train] epoch=3 step=2450/3145 loss=0.0938 loss_action=0.0034 loss_video=0.0904 lr=1.36e-05 speed=0.02 step/s, 2.45 samples/s eta=10:03:56 |
| [2026-07-20 19:23:41,792][fastwam.trainer][INFO] - [train] epoch=3 step=2460/3145 loss=0.0998 loss_action=0.0038 loss_video=0.0960 lr=1.33e-05 speed=0.02 step/s, 2.46 samples/s eta=09:55:12 |
| [2026-07-20 19:32:12,657][fastwam.trainer][INFO] - [train] epoch=3 step=2470/3145 loss=0.0912 loss_action=0.0060 loss_video=0.0852 lr=1.30e-05 speed=0.02 step/s, 2.46 samples/s eta=09:46:28 |
| [2026-07-20 19:40:42,738][fastwam.trainer][INFO] - [train] epoch=3 step=2480/3145 loss=0.0905 loss_action=0.0034 loss_video=0.0872 lr=1.26e-05 speed=0.02 step/s, 2.46 samples/s eta=09:37:43 |
| [2026-07-20 19:49:14,509][fastwam.trainer][INFO] - [train] epoch=3 step=2490/3145 loss=0.1089 loss_action=0.0046 loss_video=0.1043 lr=1.23e-05 speed=0.02 step/s, 2.46 samples/s eta=09:29:00 |
| [2026-07-20 19:57:48,046][fastwam.trainer][INFO] - [train] epoch=3 step=2500/3145 loss=0.1048 loss_action=0.0036 loss_video=0.1012 lr=1.20e-05 speed=0.02 step/s, 2.46 samples/s eta=09:20:16 |
| [2026-07-20 20:06:20,544][fastwam.trainer][INFO] - [train] epoch=3 step=2510/3145 loss=0.0911 loss_action=0.0034 loss_video=0.0877 lr=1.16e-05 speed=0.02 step/s, 2.46 samples/s eta=09:11:33 |
| [2026-07-20 20:14:54,667][fastwam.trainer][INFO] - [train] epoch=4 step=2520/3145 loss=0.1003 loss_action=0.0041 loss_video=0.0962 lr=1.13e-05 speed=0.02 step/s, 2.46 samples/s eta=09:02:50 |
| [2026-07-20 20:23:25,374][fastwam.trainer][INFO] - [train] epoch=4 step=2530/3145 loss=0.0977 loss_action=0.0049 loss_video=0.0927 lr=1.10e-05 speed=0.02 step/s, 2.46 samples/s eta=08:54:06 |
| [2026-07-20 20:31:58,437][fastwam.trainer][INFO] - [train] epoch=4 step=2540/3145 loss=0.0927 loss_action=0.0053 loss_video=0.0873 lr=1.07e-05 speed=0.02 step/s, 2.46 samples/s eta=08:45:23 |
| [2026-07-20 20:40:30,838][fastwam.trainer][INFO] - [train] epoch=4 step=2550/3145 loss=0.1045 loss_action=0.0055 loss_video=0.0990 lr=1.04e-05 speed=0.02 step/s, 2.46 samples/s eta=08:36:40 |
| [2026-07-20 20:49:00,921][fastwam.trainer][INFO] - [train] epoch=4 step=2560/3145 loss=0.0831 loss_action=0.0053 loss_video=0.0778 lr=1.01e-05 speed=0.02 step/s, 2.46 samples/s eta=08:27:57 |
| [2026-07-20 20:57:34,426][fastwam.trainer][INFO] - [train] epoch=4 step=2570/3145 loss=0.1018 loss_action=0.0037 loss_video=0.0982 lr=9.77e-06 speed=0.02 step/s, 2.46 samples/s eta=08:19:14 |
| [2026-07-20 21:06:04,663][fastwam.trainer][INFO] - [train] epoch=4 step=2580/3145 loss=0.0909 loss_action=0.0035 loss_video=0.0874 lr=9.48e-06 speed=0.02 step/s, 2.46 samples/s eta=08:10:31 |
| [2026-07-20 21:14:34,321][fastwam.trainer][INFO] - [train] epoch=4 step=2590/3145 loss=0.0951 loss_action=0.0047 loss_video=0.0904 lr=9.19e-06 speed=0.02 step/s, 2.46 samples/s eta=08:01:47 |
| [2026-07-20 21:23:05,329][fastwam.trainer][INFO] - [train] epoch=4 step=2600/3145 loss=0.0979 loss_action=0.0045 loss_video=0.0933 lr=8.91e-06 speed=0.02 step/s, 2.46 samples/s eta=07:53:05 |
| [2026-07-20 21:31:38,507][fastwam.trainer][INFO] - [train] epoch=4 step=2610/3145 loss=0.0873 loss_action=0.0049 loss_video=0.0824 lr=8.63e-06 speed=0.02 step/s, 2.46 samples/s eta=07:44:22 |
| [2026-07-20 21:40:08,618][fastwam.trainer][INFO] - [train] epoch=4 step=2620/3145 loss=0.0886 loss_action=0.0027 loss_video=0.0859 lr=8.35e-06 speed=0.02 step/s, 2.46 samples/s eta=07:35:39 |
| [2026-07-20 21:48:36,066][fastwam.trainer][INFO] - [train] epoch=4 step=2630/3145 loss=0.0834 loss_action=0.0048 loss_video=0.0786 lr=8.08e-06 speed=0.02 step/s, 2.46 samples/s eta=07:26:56 |
| [2026-07-20 21:57:05,731][fastwam.trainer][INFO] - [train] epoch=4 step=2640/3145 loss=0.1084 loss_action=0.0046 loss_video=0.1038 lr=7.82e-06 speed=0.02 step/s, 2.46 samples/s eta=07:18:13 |
| [2026-07-20 22:05:35,521][fastwam.trainer][INFO] - [train] epoch=4 step=2650/3145 loss=0.0995 loss_action=0.0041 loss_video=0.0954 lr=7.55e-06 speed=0.02 step/s, 2.46 samples/s eta=07:09:30 |
| [2026-07-20 22:14:04,756][fastwam.trainer][INFO] - [train] epoch=4 step=2660/3145 loss=0.0838 loss_action=0.0046 loss_video=0.0793 lr=7.30e-06 speed=0.02 step/s, 2.46 samples/s eta=07:00:48 |
| [2026-07-20 22:22:33,443][fastwam.trainer][INFO] - [train] epoch=4 step=2670/3145 loss=0.1059 loss_action=0.0049 loss_video=0.1009 lr=7.05e-06 speed=0.02 step/s, 2.46 samples/s eta=06:52:05 |
| [2026-07-20 22:31:02,870][fastwam.trainer][INFO] - [train] epoch=4 step=2680/3145 loss=0.0979 loss_action=0.0031 loss_video=0.0948 lr=6.80e-06 speed=0.02 step/s, 2.46 samples/s eta=06:43:22 |
| [2026-07-20 22:39:32,344][fastwam.trainer][INFO] - [train] epoch=4 step=2690/3145 loss=0.0923 loss_action=0.0033 loss_video=0.0890 lr=6.56e-06 speed=0.02 step/s, 2.46 samples/s eta=06:34:40 |
| [2026-07-20 22:47:59,174][fastwam.trainer][INFO] - [train] epoch=4 step=2700/3145 loss=0.1007 loss_action=0.0031 loss_video=0.0977 lr=6.32e-06 speed=0.02 step/s, 2.46 samples/s eta=06:25:57 |
| [2026-07-20 22:56:27,760][fastwam.trainer][INFO] - [train] epoch=4 step=2710/3145 loss=0.1066 loss_action=0.0035 loss_video=0.1032 lr=6.09e-06 speed=0.02 step/s, 2.46 samples/s eta=06:17:15 |
| [2026-07-20 23:04:57,687][fastwam.trainer][INFO] - [train] epoch=4 step=2720/3145 loss=0.0886 loss_action=0.0035 loss_video=0.0851 lr=5.86e-06 speed=0.02 step/s, 2.46 samples/s eta=06:08:33 |
| [2026-07-20 23:13:27,356][fastwam.trainer][INFO] - [train] epoch=4 step=2730/3145 loss=0.0896 loss_action=0.0033 loss_video=0.0864 lr=5.64e-06 speed=0.02 step/s, 2.46 samples/s eta=05:59:51 |
| [2026-07-20 23:22:01,231][fastwam.trainer][INFO] - [train] epoch=4 step=2740/3145 loss=0.0891 loss_action=0.0026 loss_video=0.0865 lr=5.42e-06 speed=0.02 step/s, 2.46 samples/s eta=05:51:10 |
| [2026-07-20 23:30:32,876][fastwam.trainer][INFO] - [train] epoch=4 step=2750/3145 loss=0.0962 loss_action=0.0026 loss_video=0.0936 lr=5.21e-06 speed=0.02 step/s, 2.46 samples/s eta=05:42:28 |
| [2026-07-20 23:39:02,970][fastwam.trainer][INFO] - [train] epoch=4 step=2760/3145 loss=0.0904 loss_action=0.0029 loss_video=0.0875 lr=5.00e-06 speed=0.02 step/s, 2.46 samples/s eta=05:33:47 |
| [2026-07-20 23:47:30,943][fastwam.trainer][INFO] - [train] epoch=4 step=2770/3145 loss=0.0928 loss_action=0.0051 loss_video=0.0876 lr=4.80e-06 speed=0.02 step/s, 2.46 samples/s eta=05:25:05 |
| [2026-07-20 23:56:02,345][fastwam.trainer][INFO] - [train] epoch=4 step=2780/3145 loss=0.0949 loss_action=0.0032 loss_video=0.0917 lr=4.60e-06 speed=0.02 step/s, 2.46 samples/s eta=05:16:24 |
| [2026-07-21 00:04:33,467][fastwam.trainer][INFO] - [train] epoch=4 step=2790/3145 loss=0.1161 loss_action=0.0028 loss_video=0.1133 lr=4.41e-06 speed=0.02 step/s, 2.46 samples/s eta=05:07:42 |
| [2026-07-21 00:13:03,767][fastwam.trainer][INFO] - [train] epoch=4 step=2800/3145 loss=0.0999 loss_action=0.0028 loss_video=0.0971 lr=4.22e-06 speed=0.02 step/s, 2.46 samples/s eta=04:59:01 |
| [2026-07-21 00:21:34,497][fastwam.trainer][INFO] - [train] epoch=4 step=2810/3145 loss=0.0941 loss_action=0.0031 loss_video=0.0910 lr=4.04e-06 speed=0.02 step/s, 2.46 samples/s eta=04:50:20 |
| [2026-07-21 00:30:07,603][fastwam.trainer][INFO] - [train] epoch=4 step=2820/3145 loss=0.0937 loss_action=0.0034 loss_video=0.0903 lr=3.86e-06 speed=0.02 step/s, 2.46 samples/s eta=04:41:39 |
| [2026-07-21 00:38:38,442][fastwam.trainer][INFO] - [train] epoch=4 step=2830/3145 loss=0.0821 loss_action=0.0029 loss_video=0.0793 lr=3.69e-06 speed=0.02 step/s, 2.46 samples/s eta=04:32:58 |
| [2026-07-21 00:47:08,510][fastwam.trainer][INFO] - [train] epoch=4 step=2840/3145 loss=0.1041 loss_action=0.0038 loss_video=0.1003 lr=3.52e-06 speed=0.02 step/s, 2.46 samples/s eta=04:24:17 |
| [2026-07-21 00:55:38,538][fastwam.trainer][INFO] - [train] epoch=4 step=2850/3145 loss=0.0971 loss_action=0.0024 loss_video=0.0947 lr=3.36e-06 speed=0.02 step/s, 2.46 samples/s eta=04:15:36 |
| [2026-07-21 01:04:10,892][fastwam.trainer][INFO] - [train] epoch=4 step=2860/3145 loss=0.0846 loss_action=0.0027 loss_video=0.0818 lr=3.21e-06 speed=0.02 step/s, 2.46 samples/s eta=04:06:56 |
| [2026-07-21 01:12:42,664][fastwam.trainer][INFO] - [train] epoch=4 step=2870/3145 loss=0.1009 loss_action=0.0035 loss_video=0.0974 lr=3.05e-06 speed=0.02 step/s, 2.46 samples/s eta=03:58:15 |
| [2026-07-21 01:21:15,274][fastwam.trainer][INFO] - [train] epoch=4 step=2880/3145 loss=0.1026 loss_action=0.0022 loss_video=0.1004 lr=2.91e-06 speed=0.02 step/s, 2.46 samples/s eta=03:49:34 |
| [2026-07-21 01:29:46,111][fastwam.trainer][INFO] - [train] epoch=4 step=2890/3145 loss=0.1041 loss_action=0.0025 loss_video=0.1016 lr=2.77e-06 speed=0.02 step/s, 2.46 samples/s eta=03:40:54 |
| [2026-07-21 01:38:15,502][fastwam.trainer][INFO] - [train] epoch=4 step=2900/3145 loss=0.0974 loss_action=0.0025 loss_video=0.0949 lr=2.63e-06 speed=0.02 step/s, 2.46 samples/s eta=03:32:13 |
| [2026-07-21 01:46:45,346][fastwam.trainer][INFO] - [train] epoch=4 step=2910/3145 loss=0.0821 loss_action=0.0029 loss_video=0.0792 lr=2.50e-06 speed=0.02 step/s, 2.46 samples/s eta=03:23:33 |
| [2026-07-21 01:55:13,289][fastwam.trainer][INFO] - [train] epoch=4 step=2920/3145 loss=0.0783 loss_action=0.0028 loss_video=0.0755 lr=2.38e-06 speed=0.02 step/s, 2.46 samples/s eta=03:14:52 |
| [2026-07-21 02:03:40,513][fastwam.trainer][INFO] - [train] epoch=4 step=2930/3145 loss=0.0951 loss_action=0.0023 loss_video=0.0928 lr=2.26e-06 speed=0.02 step/s, 2.46 samples/s eta=03:06:11 |
| [2026-07-21 02:12:12,299][fastwam.trainer][INFO] - [train] epoch=4 step=2940/3145 loss=0.1069 loss_action=0.0024 loss_video=0.1046 lr=2.15e-06 speed=0.02 step/s, 2.46 samples/s eta=02:57:31 |
| [2026-07-21 02:20:44,150][fastwam.trainer][INFO] - [train] epoch=4 step=2950/3145 loss=0.0898 loss_action=0.0031 loss_video=0.0867 lr=2.04e-06 speed=0.02 step/s, 2.46 samples/s eta=02:48:51 |
| [2026-07-21 02:29:16,087][fastwam.trainer][INFO] - [train] epoch=4 step=2960/3145 loss=0.0877 loss_action=0.0036 loss_video=0.0841 lr=1.93e-06 speed=0.02 step/s, 2.46 samples/s eta=02:40:11 |
| [2026-07-21 02:37:45,330][fastwam.trainer][INFO] - [train] epoch=4 step=2970/3145 loss=0.1128 loss_action=0.0022 loss_video=0.1106 lr=1.84e-06 speed=0.02 step/s, 2.46 samples/s eta=02:31:31 |
| [2026-07-21 02:46:15,417][fastwam.trainer][INFO] - [train] epoch=4 step=2980/3145 loss=0.0974 loss_action=0.0035 loss_video=0.0939 lr=1.74e-06 speed=0.02 step/s, 2.46 samples/s eta=02:22:51 |
| [2026-07-21 02:54:45,308][fastwam.trainer][INFO] - [train] epoch=4 step=2990/3145 loss=0.1003 loss_action=0.0032 loss_video=0.0971 lr=1.66e-06 speed=0.02 step/s, 2.46 samples/s eta=02:14:11 |
| [2026-07-21 03:03:16,821][fastwam.trainer][INFO] - [train] epoch=4 step=3000/3145 loss=0.0891 loss_action=0.0022 loss_video=0.0869 lr=1.57e-06 speed=0.02 step/s, 2.46 samples/s eta=02:05:31 |
| [2026-07-21 03:11:47,360][fastwam.trainer][INFO] - [train] epoch=4 step=3010/3145 loss=0.0842 loss_action=0.0026 loss_video=0.0816 lr=1.50e-06 speed=0.02 step/s, 2.46 samples/s eta=01:56:51 |
| [2026-07-21 03:20:15,867][fastwam.trainer][INFO] - [train] epoch=4 step=3020/3145 loss=0.0845 loss_action=0.0032 loss_video=0.0813 lr=1.43e-06 speed=0.02 step/s, 2.46 samples/s eta=01:48:11 |
| [2026-07-21 03:28:43,634][fastwam.trainer][INFO] - [train] epoch=4 step=3030/3145 loss=0.1086 loss_action=0.0025 loss_video=0.1061 lr=1.36e-06 speed=0.02 step/s, 2.46 samples/s eta=01:39:32 |
| [2026-07-21 03:37:11,024][fastwam.trainer][INFO] - [train] epoch=4 step=3040/3145 loss=0.1045 loss_action=0.0025 loss_video=0.1019 lr=1.30e-06 speed=0.02 step/s, 2.46 samples/s eta=01:30:52 |
| [2026-07-21 03:45:37,783][fastwam.trainer][INFO] - [train] epoch=4 step=3050/3145 loss=0.0870 loss_action=0.0022 loss_video=0.0848 lr=1.25e-06 speed=0.02 step/s, 2.47 samples/s eta=01:22:12 |
| [2026-07-21 03:54:04,190][fastwam.trainer][INFO] - [train] epoch=4 step=3060/3145 loss=0.1132 loss_action=0.0035 loss_video=0.1096 lr=1.20e-06 speed=0.02 step/s, 2.47 samples/s eta=01:13:33 |
| [2026-07-21 04:02:33,918][fastwam.trainer][INFO] - [train] epoch=4 step=3070/3145 loss=0.1070 loss_action=0.0036 loss_video=0.1034 lr=1.15e-06 speed=0.02 step/s, 2.47 samples/s eta=01:04:53 |
| [2026-07-21 04:11:01,655][fastwam.trainer][INFO] - [train] epoch=4 step=3080/3145 loss=0.0954 loss_action=0.0029 loss_video=0.0925 lr=1.12e-06 speed=0.02 step/s, 2.47 samples/s eta=00:56:14 |
| [2026-07-21 04:19:29,574][fastwam.trainer][INFO] - [train] epoch=4 step=3090/3145 loss=0.0981 loss_action=0.0025 loss_video=0.0956 lr=1.08e-06 speed=0.02 step/s, 2.47 samples/s eta=00:47:35 |
| [2026-07-21 04:27:55,731][fastwam.trainer][INFO] - [train] epoch=4 step=3100/3145 loss=0.0961 loss_action=0.0028 loss_video=0.0933 lr=1.06e-06 speed=0.02 step/s, 2.47 samples/s eta=00:38:55 |
| [2026-07-21 04:36:23,109][fastwam.trainer][INFO] - [train] epoch=4 step=3110/3145 loss=0.0911 loss_action=0.0025 loss_video=0.0886 lr=1.03e-06 speed=0.02 step/s, 2.47 samples/s eta=00:30:16 |
| [2026-07-21 04:44:51,256][fastwam.trainer][INFO] - [train] epoch=4 step=3120/3145 loss=0.1007 loss_action=0.0030 loss_video=0.0977 lr=1.02e-06 speed=0.02 step/s, 2.47 samples/s eta=00:21:37 |
| [2026-07-21 04:53:19,239][fastwam.trainer][INFO] - [train] epoch=4 step=3130/3145 loss=0.0897 loss_action=0.0034 loss_video=0.0863 lr=1.01e-06 speed=0.02 step/s, 2.47 samples/s eta=00:12:58 |
| [2026-07-21 05:01:47,763][fastwam.trainer][INFO] - [train] epoch=4 step=3140/3145 loss=0.0828 loss_action=0.0030 loss_video=0.0798 lr=1.00e-06 speed=0.02 step/s, 2.47 samples/s eta=00:04:19 |
| [2026-07-21 05:05:42,416][accelerate.accelerator][INFO] - Saving current state to /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145 |
| [2026-07-21 05:05:42,420][accelerate.accelerator][INFO] - Saving DeepSpeed Model and Optimizer |
| [2026-07-21 05:07:07,093][accelerate.accelerator][INFO] - DeepSpeed Model and Optimizer saved to output dir /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145/pytorch_model |
| [2026-07-21 05:07:07,298][accelerate.checkpointing][INFO] - Scheduler state saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145/scheduler.bin |
| [2026-07-21 05:07:07,299][accelerate.checkpointing][INFO] - Sampler state for dataloader 0 saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145/sampler.bin |
| [2026-07-21 05:07:07,355][accelerate.checkpointing][INFO] - Random states saved in /horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145/random_states_0.pkl |
| [2026-07-21 05:07:07,399][fastwam.trainer][INFO] - [done] max_steps reached step=3145 weights=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/weights/step_003145.pt state=/horizon-bucket/robot_lab/users/wenkang.hu-labs/projects/deformable_bench/runs/flatten-silk/fastwam/production-20260718T232050Z/checkpoints/state/step_003145 |
|
|
