| ==> Sat Jun 20 13:47:14 UTC 2026 start |
| NVIDIA A100-SXM4-80GB, 81920 MiB |
| Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. |
| WARNING: The scripts hf, huggingface-cli and tiny-agents are installed in '/home/ubuntu/.local/bin' which is not on PATH. |
| Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. |
| ==> Sat Jun 20 13:47:25 UTC 2026 pull BCI from HF |
| BCI tar: /tmp/bci_data.tar |
| train pairs: 3896 |
| X_1: torch.Size([3, 256, 256]) torch.float32 [-0.953, 0.718] |
| X_0: torch.Size([3, 256, 256]) torch.float32 [-0.129, 1.000] |
| ==> Sat Jun 20 13:47:51 UTC 2026 TRAIN dual-task X_0 + rotation aug |
| [ddpm] T=1000 betas=(0.0001,0.02) alpha_bar[0]=0.999900 alpha_bar[-1]=4.035829e-05 |
| [data] train=3896 HE/IHC paired, augment=True |
| [model] params=130.50M in=6 out=3 (X_0-pred, DDPM) |
| [schedule] he_gen_prob=0.5 (0.5 = HistDiST 50/50, 0.0 = single-task baseline) |
| [iter 1/3000] loss=0.3243 L_he_ema=0.0000 L_ihc_ema=0.3243 frac_he=0.000 |conv_in grad: HE=0.00e+00 IHC=0.00e+00 ratio_I/H=0.00| wall=0.1min |
| [iter 100/3000] loss=0.0433 L_he_ema=0.0659 L_ihc_ema=0.0819 frac_he=0.535 |conv_in grad: HE=6.48e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=2.5min |
| [iter 200/3000] loss=0.0970 L_he_ema=0.0549 L_ihc_ema=0.0727 frac_he=0.580 |conv_in grad: HE=1.21e-01 IHC=7.29e-02 ratio_I/H=0.60| wall=5.0min |
| [iter 300/3000] loss=0.0266 L_he_ema=0.0557 L_ihc_ema=0.0435 frac_he=0.510 |conv_in grad: HE=5.90e-02 IHC=5.89e-02 ratio_I/H=1.00| wall=7.5min |
| [iter 400/3000] loss=0.0518 L_he_ema=0.0518 L_ihc_ema=0.0472 frac_he=0.400 |conv_in grad: HE=2.02e-01 IHC=1.89e-01 ratio_I/H=0.93| wall=10.0min |
| [iter 500/3000] loss=0.0473 L_he_ema=0.0524 L_ihc_ema=0.0533 frac_he=0.520 |conv_in grad: HE=5.30e-02 IHC=5.74e-02 ratio_I/H=1.08| wall=12.5min |
| [iter 600/3000] loss=0.0398 L_he_ema=0.0452 L_ihc_ema=0.0396 frac_he=0.530 |conv_in grad: HE=5.76e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=15.3min |
| [iter 700/3000] loss=0.0404 L_he_ema=0.0477 L_ihc_ema=0.0358 frac_he=0.510 |conv_in grad: HE=8.11e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=17.9min |
| [iter 800/3000] loss=0.0495 L_he_ema=0.0567 L_ihc_ema=0.0415 frac_he=0.480 |conv_in grad: HE=1.45e-01 IHC=0.00e+00 ratio_I/H=0.00| wall=20.6min |
| [iter 900/3000] loss=0.0306 L_he_ema=0.0458 L_ihc_ema=0.0321 frac_he=0.500 |conv_in grad: HE=1.99e-01 IHC=1.17e-01 ratio_I/H=0.59| wall=23.1min |
| [iter 1000/3000] loss=0.0425 L_he_ema=0.0618 L_ihc_ema=0.0466 frac_he=0.480 |conv_in grad: HE=6.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=25.6min |
| [save] iter 1000 |
| [iter 1100/3000] loss=0.0342 L_he_ema=0.0417 L_ihc_ema=0.0385 frac_he=0.540 |conv_in grad: HE=1.91e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=28.5min |
| [iter 1200/3000] loss=0.0185 L_he_ema=0.0497 L_ihc_ema=0.0428 frac_he=0.550 |conv_in grad: HE=4.63e-02 IHC=4.81e-02 ratio_I/H=1.04| wall=31.3min |
| [iter 1300/3000] loss=0.0213 L_he_ema=0.0465 L_ihc_ema=0.0388 frac_he=0.570 |conv_in grad: HE=7.59e-02 IHC=3.32e-02 ratio_I/H=0.44| wall=33.9min |
| [iter 1400/3000] loss=0.0495 L_he_ema=0.0439 L_ihc_ema=0.0390 frac_he=0.540 |conv_in grad: HE=1.07e-01 IHC=0.00e+00 ratio_I/H=0.00| wall=36.4min |
| [iter 1500/3000] loss=0.0346 L_he_ema=0.0412 L_ihc_ema=0.0398 frac_he=0.580 |conv_in grad: HE=7.32e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=39.0min |
| [iter 1600/3000] loss=0.0277 L_he_ema=0.0380 L_ihc_ema=0.0272 frac_he=0.490 |conv_in grad: HE=9.41e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=41.7min |
| [iter 1700/3000] loss=0.0406 L_he_ema=0.0500 L_ihc_ema=0.0266 frac_he=0.460 |conv_in grad: HE=3.25e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=44.3min |
| [iter 1800/3000] loss=0.0179 L_he_ema=0.0426 L_ihc_ema=0.0313 frac_he=0.550 |conv_in grad: HE=2.29e-02 IHC=2.75e-02 ratio_I/H=1.20| wall=46.7min |
| [iter 1900/3000] loss=0.0278 L_he_ema=0.0395 L_ihc_ema=0.0303 frac_he=0.490 |conv_in grad: HE=4.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=49.2min |
| [iter 2000/3000] loss=0.0122 L_he_ema=0.0337 L_ihc_ema=0.0228 frac_he=0.560 |conv_in grad: HE=5.52e-02 IHC=3.47e-02 ratio_I/H=0.63| wall=51.8min |
| [save] iter 2000 |
| [iter 2100/3000] loss=0.0365 L_he_ema=0.0459 L_ihc_ema=0.0349 frac_he=0.530 |conv_in grad: HE=4.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=54.7min |
| [iter 2200/3000] loss=0.0192 L_he_ema=0.0406 L_ihc_ema=0.0273 frac_he=0.520 |conv_in grad: HE=3.05e-02 IHC=3.19e-02 ratio_I/H=1.05| wall=57.2min |
| [iter 2300/3000] loss=0.0396 L_he_ema=0.0389 L_ihc_ema=0.0286 frac_he=0.440 |conv_in grad: HE=4.42e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=59.7min |
| [iter 2400/3000] loss=0.0231 L_he_ema=0.0355 L_ihc_ema=0.0253 frac_he=0.480 |conv_in grad: HE=8.65e-02 IHC=8.11e-02 ratio_I/H=0.94| wall=62.3min |
| [iter 2500/3000] loss=0.0450 L_he_ema=0.0381 L_ihc_ema=0.0259 frac_he=0.540 |conv_in grad: HE=6.44e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=64.8min |
| [iter 2600/3000] loss=0.0372 L_he_ema=0.0448 L_ihc_ema=0.0283 frac_he=0.450 |conv_in grad: HE=6.93e-02 IHC=6.99e-02 ratio_I/H=1.01| wall=67.6min |
| [iter 2700/3000] loss=0.0358 L_he_ema=0.0365 L_ihc_ema=0.0292 frac_he=0.510 |conv_in grad: HE=5.75e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=70.1min |
| [iter 2800/3000] loss=0.0494 L_he_ema=0.0458 L_ihc_ema=0.0303 frac_he=0.640 |conv_in grad: HE=1.86e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=72.7min |
| [iter 2900/3000] loss=0.0212 L_he_ema=0.0324 L_ihc_ema=0.0240 frac_he=0.480 |conv_in grad: HE=3.67e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=75.2min |
| [iter 3000/3000] loss=0.0333 L_he_ema=0.0389 L_ihc_ema=0.0274 frac_he=0.460 |conv_in grad: HE=1.32e-01 IHC=1.54e-01 ratio_I/H=1.17| wall=77.8min |
| [save] iter 3000 |
| [done] wall=78.1min |
| ==> Sat Jun 20 15:06:15 UTC 2026 SAMPLE + FID NFE=100 |
| [ckpt] iter=3000 he_gen_prob=0.5 |
| [ema applied] |
| [data] n_test=977 batch=16 |
| [sampled] 64/977 |
| [sampled] 128/977 |
| [sampled] 192/977 |
| [sampled] 256/977 |
| [sampled] 320/977 |
| [sampled] 384/977 |
| [sampled] 448/977 |
| [sampled] 512/977 |
| [sampled] 576/977 |
| [sampled] 640/977 |
| [sampled] 704/977 |
| [sampled] 768/977 |
| [sampled] 832/977 |
| [sampled] 896/977 |
| [sampled] 960/977 |
| [sampled] 977/977 |
| [done] 977 samples saved |
| compute FID between two folders |
| Found 977 images in the folder /home/ubuntu/work/dualtask_rot_fid/samples |
|
FID samples : 0%| | 0/16 [00:00<?, ?it/s]
FID samples : 6%|▋ | 1/16 [00:01<00:24, 1.64s/it]
FID samples : 12%|█▎ | 2/16 [00:06<00:46, 3.31s/it]
FID samples : 19%|█▉ | 3/16 [00:06<00:24, 1.91s/it]
FID samples : 25%|██▌ | 4/16 [00:06<00:14, 1.24s/it]
FID samples : 31%|███▏ | 5/16 [00:06<00:09, 1.15it/s]
FID samples : 38%|███▊ | 6/16 [00:07<00:06, 1.54it/s]
FID samples : 44%|████▍ | 7/16 [00:07<00:04, 1.96it/s]
FID samples : 50%|█████ | 8/16 [00:07<00:03, 2.42it/s]
FID samples : 56%|█████▋ | 9/16 [00:07<00:02, 2.79it/s]
FID samples : 62%|██████▎ | 10/16 [00:07<00:01, 3.01it/s]
FID samples : 69%|██████▉ | 11/16 [00:08<00:01, 3.14it/s]
FID samples : 75%|███████▌ | 12/16 [00:08<00:01, 3.32it/s]
FID samples : 81%|████████▏ | 13/16 [00:08<00:00, 3.61it/s]
FID samples : 88%|████████▊ | 14/16 [00:08<00:00, 3.67it/s]
FID samples : 94%|█████████▍| 15/16 [00:09<00:00, 3.77it/s]
FID samples : 100%|██████████| 16/16 [00:09<00:00, 3.53it/s]
FID samples : 100%|██████████| 16/16 [00:09<00:00, 1.65it/s] |
| Found 977 images in the folder /home/ubuntu/work/dualtask_rot_fid/target_ihc |
|
FID target_ihc : 0%| | 0/16 [00:00<?, ?it/s]
FID target_ihc : 6%|▋ | 1/16 [00:01<00:17, 1.18s/it]
FID target_ihc : 12%|█▎ | 2/16 [00:01<00:08, 1.59it/s]
FID target_ihc : 19%|█▉ | 3/16 [00:01<00:06, 1.92it/s]
FID target_ihc : 25%|██▌ | 4/16 [00:02<00:04, 2.45it/s]
FID target_ihc : 31%|███▏ | 5/16 [00:02<00:04, 2.54it/s]
FID target_ihc : 38%|███▊ | 6/16 [00:02<00:03, 2.97it/s]
FID target_ihc : 44%|████▍ | 7/16 [00:02<00:03, 2.94it/s]
FID target_ihc : 50%|█████ | 8/16 [00:03<00:02, 3.23it/s]
FID target_ihc : 56%|█████▋ | 9/16 [00:03<00:02, 3.06it/s]
FID target_ihc : 62%|██████▎ | 10/16 [00:03<00:01, 3.22it/s]
FID target_ihc : 69%|██████▉ | 11/16 [00:04<00:01, 3.28it/s]
FID target_ihc : 75%|███████▌ | 12/16 [00:04<00:01, 3.54it/s]
FID target_ihc : 81%|████████▏ | 13/16 [00:04<00:00, 3.16it/s]
FID target_ihc : 88%|████████▊ | 14/16 [00:05<00:00, 3.48it/s]
FID target_ihc : 94%|█████████▍| 15/16 [00:05<00:00, 3.26it/s]
FID target_ihc : 100%|██████████| 16/16 [00:08<00:00, 1.12s/it]
FID target_ihc : 100%|██████████| 16/16 [00:08<00:00, 1.87it/s] |
| FID (clean-fid, NFE=100) = 267.235 |
| ==> Sat Jun 20 15:24:24 UTC 2026 HF push dualtask |
|
Processing Files (0 / 0) : | | 0.00B / 0.00B |
|
New Data Upload : | | 0.00B / 0.00B [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A
Processing Files (0 / 1) : 0%| | 80.1kB / 2.09GB, 7.87kB/s |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 80.1kB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 640kB / 2.09GB [A[A
Processing Files (0 / 1) : 0%| | 640kB / 2.09GB, 61.4kB/s |
|
New Data Upload : 1%| | 560kB / 67.1MB, 54.0kB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 1.20MB / 2.09GB [A[A
Processing Files (0 / 1) : 0%| | 1.20MB / 2.09GB, 115kB/s |
|
New Data Upload : 1%| | 1.12MB / 134MB, 107kB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 4.56MB / 2.09GB [A[A
Processing Files (0 / 1) : 0%| | 4.56MB / 2.09GB, 436kB/s |
|
New Data Upload : 3%|▎ | 4.48MB / 134MB, 429kB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 0%| | 10.2MB / 2.09GB [A[A
Processing Files (0 / 1) : 0%| | 10.2MB / 2.09GB, 968kB/s |
|
New Data Upload : 8%|▊ | 10.1MB / 134MB, 960kB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 1%| | 18.6MB / 2.09GB [A[A
Processing Files (0 / 1) : 1%| | 18.6MB / 2.09GB, 1.76MB/s |
|
New Data Upload : 14%|█▍ | 18.5MB / 134MB, 1.75MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 1%|▏ | 30.9MB / 2.09GB [A[A
Processing Files (0 / 1) : 1%|▏ | 30.9MB / 2.09GB, 2.92MB/s |
|
New Data Upload : 23%|██▎ | 30.8MB / 134MB, 2.91MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 2%|▏ | 42.1MB / 2.09GB [A[A
Processing Files (0 / 1) : 2%|▏ | 42.1MB / 2.09GB, 3.95MB/s |
|
New Data Upload : 31%|███▏ | 42.0MB / 134MB, 3.94MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 3%|▎ | 53.8MB / 2.09GB [A[A
Processing Files (0 / 1) : 3%|▎ | 53.8MB / 2.09GB, 5.03MB/s |
|
New Data Upload : 27%|██▋ | 53.7MB / 201MB, 5.02MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 3%|▎ | 71.1MB / 2.09GB [A[A
Processing Files (0 / 1) : 3%|▎ | 71.1MB / 2.09GB, 6.62MB/s |
|
New Data Upload : 35%|███▌ | 71.1MB / 201MB, 6.61MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 4%|▍ | 89.6MB / 2.09GB [A[A
Processing Files (0 / 1) : 4%|▍ | 89.6MB / 2.09GB, 8.29MB/s |
|
New Data Upload : 45%|████▍ | 89.5MB / 201MB, 8.28MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 5%|▍ | 99.6MB / 2.09GB [A[A
Processing Files (0 / 1) : 5%|▍ | 99.6MB / 2.09GB, 9.16MB/s |
|
New Data Upload : 50%|████▉ | 99.6MB / 201MB, 9.15MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 5%|▌ | 110MB / 2.09GB [A[A
Processing Files (0 / 1) : 5%|▌ | 110MB / 2.09GB, 10.1MB/s |
|
New Data Upload : 55%|█████▍ | 110MB / 201MB, 10.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▌ | 121MB / 2.09GB [A[A
Processing Files (0 / 1) : 6%|▌ | 121MB / 2.09GB, 11.0MB/s |
|
New Data Upload : 60%|██████ | 121MB / 201MB, 11.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▌ | 130MB / 2.09GB [A[A
Processing Files (0 / 1) : 6%|▌ | 130MB / 2.09GB, 11.7MB/s |
|
New Data Upload : 64%|██████▍ | 130MB / 201MB, 11.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▋ | 134MB / 2.09GB [A[A
Processing Files (0 / 1) : 6%|▋ | 134MB / 2.09GB, 11.9MB/s |
|
New Data Upload : 66%|██████▋ | 134MB / 201MB, 11.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▋ | 134MB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▋ | 134MB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 6%|▋ | 134MB / 2.09GB [A[A
Processing Files (0 / 1) : 6%|▋ | 134MB / 2.09GB, 11.7MB/s |
|
New Data Upload : 67%|██████▋ | 134MB / 201MB, 11.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 7%|▋ | 139MB / 2.09GB [A[A
Processing Files (0 / 1) : 7%|▋ | 139MB / 2.09GB, 12.0MB/s |
|
New Data Upload : 69%|██████▉ | 139MB / 201MB, 12.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 7%|▋ | 150MB / 2.09GB [A[A
Processing Files (0 / 1) : 7%|▋ | 150MB / 2.09GB, 12.9MB/s |
|
New Data Upload : 56%|█████▌ | 150MB / 268MB, 12.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 8%|▊ | 159MB / 2.09GB [A[A
Processing Files (0 / 1) : 8%|▊ | 159MB / 2.09GB, 13.6MB/s |
|
New Data Upload : 59%|█████▉ | 159MB / 268MB, 13.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 8%|▊ | 168MB / 2.09GB [A[A
Processing Files (0 / 1) : 8%|▊ | 168MB / 2.09GB, 14.3MB/s |
|
New Data Upload : 63%|██████▎ | 168MB / 268MB, 14.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 8%|▊ | 175MB / 2.09GB [A[A
Processing Files (0 / 1) : 8%|▊ | 175MB / 2.09GB, 14.8MB/s |
|
New Data Upload : 65%|██████▌ | 175MB / 268MB, 14.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 9%|▉ | 196MB / 2.09GB [A[A
Processing Files (0 / 1) : 9%|▉ | 196MB / 2.09GB, 16.6MB/s |
|
New Data Upload : 58%|█████▊ | 196MB / 335MB, 16.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 10%|█ | 216MB / 2.09GB [A[A
Processing Files (0 / 1) : 10%|█ | 216MB / 2.09GB, 18.2MB/s |
|
New Data Upload : 64%|██████▍ | 216MB / 335MB, 18.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 11%|█ | 229MB / 2.09GB [A[A
Processing Files (0 / 1) : 11%|█ | 229MB / 2.09GB, 19.3MB/s |
|
New Data Upload : 68%|██████▊ | 229MB / 335MB, 19.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 12%|█▏ | 240MB / 2.09GB [A[A
Processing Files (0 / 1) : 12%|█▏ | 240MB / 2.09GB, 20.1MB/s |
|
New Data Upload : 72%|███████▏ | 240MB / 335MB, 20.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 12%|█▏ | 250MB / 2.09GB [A[A
Processing Files (0 / 1) : 12%|█▏ | 250MB / 2.09GB, 20.8MB/s |
|
New Data Upload : 75%|███████▍ | 250MB / 335MB, 20.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 12%|█▏ | 261MB / 2.09GB [A[A
Processing Files (0 / 1) : 12%|█▏ | 261MB / 2.09GB, 21.6MB/s |
|
New Data Upload : 65%|██████▍ | 260MB / 402MB, 21.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 13%|█▎ | 272MB / 2.09GB [A[A
Processing Files (0 / 1) : 13%|█▎ | 272MB / 2.09GB, 22.4MB/s |
|
New Data Upload : 68%|██████▊ | 272MB / 402MB, 22.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 13%|█▎ | 279MB / 2.09GB [A[A
Processing Files (0 / 1) : 13%|█▎ | 279MB / 2.09GB, 22.8MB/s |
|
New Data Upload : 69%|██████▉ | 279MB / 402MB, 22.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 14%|█▍ | 290MB / 2.09GB [A[A
Processing Files (0 / 1) : 14%|█▍ | 290MB / 2.09GB, 23.6MB/s |
|
New Data Upload : 72%|███████▏ | 290MB / 402MB, 23.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 14%|█▍ | 299MB / 2.09GB [A[A
Processing Files (0 / 1) : 14%|█▍ | 299MB / 2.09GB, 24.2MB/s |
|
New Data Upload : 74%|███████▍ | 299MB / 402MB, 24.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 15%|█▍ | 311MB / 2.09GB [A[A
Processing Files (0 / 1) : 15%|█▍ | 311MB / 2.09GB, 25.1MB/s |
|
New Data Upload : 66%|██████▋ | 311MB / 469MB, 25.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 16%|█▌ | 325MB / 2.09GB [A[A
Processing Files (0 / 1) : 16%|█▌ | 325MB / 2.09GB, 26.0MB/s |
|
New Data Upload : 69%|██████▉ | 325MB / 469MB, 26.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 16%|█▋ | 340MB / 2.09GB [A[A
Processing Files (0 / 1) : 16%|█▋ | 340MB / 2.09GB, 27.2MB/s |
|
New Data Upload : 73%|███████▎ | 340MB / 469MB, 27.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 18%|█▊ | 366MB / 2.09GB [A[A
Processing Files (0 / 1) : 18%|█▊ | 366MB / 2.09GB, 29.2MB/s |
|
New Data Upload : 78%|███████▊ | 365MB / 469MB, 29.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 18%|█▊ | 383MB / 2.09GB [A[A
Processing Files (0 / 1) : 18%|█▊ | 383MB / 2.09GB, 30.5MB/s |
|
New Data Upload : 82%|████████▏ | 383MB / 469MB, 30.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 20%|█▉ | 408MB / 2.09GB [A[A
Processing Files (0 / 1) : 20%|█▉ | 408MB / 2.09GB, 32.4MB/s |
|
New Data Upload : 76%|███████▌ | 408MB / 536MB, 32.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 20%|██ | 427MB / 2.09GB [A[A
Processing Files (0 / 1) : 20%|██ | 427MB / 2.09GB, 33.7MB/s |
|
New Data Upload : 80%|███████▉ | 427MB / 536MB, 33.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 21%|██▏ | 446MB / 2.09GB [A[A
Processing Files (0 / 1) : 21%|██▏ | 446MB / 2.09GB, 35.1MB/s |
|
New Data Upload : 83%|████████▎ | 446MB / 536MB, 35.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 22%|██▏ | 456MB / 2.09GB [A[A
Processing Files (0 / 1) : 22%|██▏ | 456MB / 2.09GB, 35.7MB/s |
|
New Data Upload : 85%|████████▌ | 456MB / 536MB, 35.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 23%|██▎ | 479MB / 2.09GB [A[A
Processing Files (0 / 1) : 23%|██▎ | 479MB / 2.09GB, 37.4MB/s |
|
New Data Upload : 89%|████████▉ | 479MB / 536MB, 37.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 23%|██▎ | 488MB / 2.09GB [A[A
Processing Files (0 / 1) : 23%|██▎ | 488MB / 2.09GB, 37.9MB/s |
|
New Data Upload : 81%|████████ | 488MB / 604MB, 37.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 24%|██▍ | 500MB / 2.09GB [A[A
Processing Files (0 / 1) : 24%|██▍ | 500MB / 2.09GB, 38.6MB/s |
|
New Data Upload : 83%|████████▎ | 500MB / 604MB, 38.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 24%|██▍ | 510MB / 2.09GB [A[A
Processing Files (0 / 1) : 24%|██▍ | 510MB / 2.09GB, 39.1MB/s |
|
New Data Upload : 85%|████████▍ | 510MB / 604MB, 39.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 25%|██▌ | 526MB / 2.09GB [A[A
Processing Files (0 / 1) : 25%|██▌ | 526MB / 2.09GB, 40.1MB/s |
|
New Data Upload : 87%|████████▋ | 525MB / 604MB, 40.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 26%|██▌ | 545MB / 2.09GB [A[A
Processing Files (0 / 1) : 26%|██▌ | 545MB / 2.09GB, 41.4MB/s |
|
New Data Upload : 81%|████████▏ | 545MB / 671MB, 41.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 27%|██▋ | 560MB / 2.09GB [A[A
Processing Files (0 / 1) : 27%|██▋ | 560MB / 2.09GB, 42.4MB/s |
|
New Data Upload : 84%|████████▎ | 560MB / 671MB, 42.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 27%|██▋ | 569MB / 2.09GB [A[A
Processing Files (0 / 1) : 27%|██▋ | 569MB / 2.09GB, 42.7MB/s |
|
New Data Upload : 85%|████████▍ | 568MB / 671MB, 42.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 28%|██▊ | 577MB / 2.09GB [A[A
Processing Files (0 / 1) : 28%|██▊ | 577MB / 2.09GB, 43.1MB/s |
|
New Data Upload : 86%|████████▌ | 577MB / 671MB, 43.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 28%|██▊ | 589MB / 2.09GB [A[A
Processing Files (0 / 1) : 28%|██▊ | 589MB / 2.09GB, 43.7MB/s |
|
New Data Upload : 80%|███████▉ | 589MB / 738MB, 43.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 29%|██▊ | 599MB / 2.09GB [A[A
Processing Files (0 / 1) : 29%|██▊ | 599MB / 2.09GB, 44.2MB/s |
|
New Data Upload : 81%|████████ | 599MB / 738MB, 44.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 29%|██▉ | 608MB / 2.09GB [A[A
Processing Files (0 / 1) : 29%|██▉ | 608MB / 2.09GB, 44.6MB/s |
|
New Data Upload : 82%|████████▏ | 608MB / 738MB, 44.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 29%|██▉ | 615MB / 2.09GB [A[A
Processing Files (0 / 1) : 29%|██▉ | 615MB / 2.09GB, 44.8MB/s |
|
New Data Upload : 83%|████████▎ | 614MB / 738MB, 44.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 30%|██▉ | 624MB / 2.09GB [A[A
Processing Files (0 / 1) : 30%|██▉ | 624MB / 2.09GB, 45.2MB/s |
|
New Data Upload : 85%|████████▍ | 624MB / 738MB, 45.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 30%|███ | 636MB / 2.09GB [A[A
Processing Files (0 / 1) : 30%|███ | 636MB / 2.09GB, 45.9MB/s |
|
New Data Upload : 79%|███████▉ | 636MB / 805MB, 45.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 31%|███ | 648MB / 2.09GB [A[A
Processing Files (0 / 1) : 31%|███ | 648MB / 2.09GB, 46.5MB/s |
|
New Data Upload : 81%|████████ | 648MB / 805MB, 46.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 32%|███▏ | 666MB / 2.09GB [A[A
Processing Files (0 / 1) : 32%|███▏ | 666MB / 2.09GB, 47.6MB/s |
|
New Data Upload : 83%|████████▎ | 666MB / 805MB, 47.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 33%|███▎ | 689MB / 2.09GB [A[A
Processing Files (0 / 1) : 33%|███▎ | 689MB / 2.09GB, 49.2MB/s |
|
New Data Upload : 86%|████████▌ | 689MB / 805MB, 49.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 35%|███▍ | 723MB / 2.09GB [A[A
Processing Files (0 / 1) : 35%|███▍ | 723MB / 2.09GB, 51.6MB/s |
|
New Data Upload : 83%|████████▎ | 723MB / 872MB, 51.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 36%|███▌ | 747MB / 2.09GB [A[A
Processing Files (0 / 1) : 36%|███▌ | 747MB / 2.09GB, 53.2MB/s |
|
New Data Upload : 86%|████████▌ | 747MB / 872MB, 53.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 37%|███▋ | 768MB / 2.09GB [A[A
Processing Files (0 / 1) : 37%|███▋ | 768MB / 2.09GB, 54.5MB/s |
|
New Data Upload : 88%|████████▊ | 768MB / 872MB, 54.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 38%|███▊ | 789MB / 2.09GB [A[A
Processing Files (0 / 1) : 38%|███▊ | 789MB / 2.09GB, 55.8MB/s |
|
New Data Upload : 90%|█████████ | 789MB / 872MB, 55.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 39%|███▊ | 809MB / 2.09GB [A[A
Processing Files (0 / 1) : 39%|███▊ | 809MB / 2.09GB, 57.0MB/s |
|
New Data Upload : 86%|████████▌ | 809MB / 939MB, 57.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 40%|███▉ | 827MB / 2.09GB [A[A
Processing Files (0 / 1) : 40%|███▉ | 827MB / 2.09GB, 58.0MB/s |
|
New Data Upload : 88%|████████▊ | 827MB / 939MB, 58.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 40%|████ | 837MB / 2.09GB [A[A
Processing Files (0 / 1) : 40%|████ | 837MB / 2.09GB, 58.4MB/s |
|
New Data Upload : 89%|████████▉ | 837MB / 939MB, 58.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 40%|████ | 845MB / 2.09GB [A[A
Processing Files (0 / 1) : 40%|████ | 845MB / 2.09GB, 58.6MB/s |
|
New Data Upload : 90%|█████████ | 845MB / 939MB, 58.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 42%|████▏ | 868MB / 2.09GB [A[A
Processing Files (0 / 1) : 42%|████▏ | 868MB / 2.09GB, 60.0MB/s |
|
New Data Upload : 92%|█████████▏| 868MB / 939MB, 60.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 42%|████▏ | 883MB / 2.09GB [A[A
Processing Files (0 / 1) : 42%|████▏ | 883MB / 2.09GB, 60.8MB/s |
|
New Data Upload : 88%|████████▊ | 883MB / 1.01GB, 60.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 43%|████▎ | 897MB / 2.09GB [A[A
Processing Files (0 / 1) : 43%|████▎ | 897MB / 2.09GB, 61.5MB/s |
|
New Data Upload : 89%|████████▉ | 897MB / 1.01GB, 61.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 43%|████▎ | 908MB / 2.09GB [A[A
Processing Files (0 / 1) : 43%|████▎ | 908MB / 2.09GB, 61.8MB/s |
|
New Data Upload : 90%|█████████ | 908MB / 1.01GB, 61.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 44%|████▍ | 921MB / 2.09GB [A[A
Processing Files (0 / 1) : 44%|████▍ | 921MB / 2.09GB, 62.4MB/s |
|
New Data Upload : 92%|█████████▏| 921MB / 1.01GB, 62.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 45%|████▌ | 942MB / 2.09GB [A[A
Processing Files (0 / 1) : 45%|████▌ | 942MB / 2.09GB, 63.6MB/s |
|
New Data Upload : 88%|████████▊ | 942MB / 1.07GB, 63.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 46%|████▌ | 963MB / 2.09GB [A[A
Processing Files (0 / 1) : 46%|████▌ | 963MB / 2.09GB, 64.9MB/s |
|
New Data Upload : 90%|████████▉ | 963MB / 1.07GB, 64.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 47%|████▋ | 975MB / 2.09GB [A[A
Processing Files (0 / 1) : 47%|████▋ | 975MB / 2.09GB, 65.2MB/s |
|
New Data Upload : 91%|█████████ | 975MB / 1.07GB, 65.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 47%|████▋ | 986MB / 2.09GB [A[A
Processing Files (0 / 1) : 47%|████▋ | 986MB / 2.09GB, 65.6MB/s |
|
New Data Upload : 92%|█████████▏| 986MB / 1.07GB, 65.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 48%|████▊ | 1.01GB / 2.09GB [A[A
Processing Files (0 / 1) : 48%|████▊ | 1.01GB / 2.09GB, 67.1MB/s |
|
New Data Upload : 89%|████████▊ | 1.01GB / 1.14GB, 67.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 49%|████▉ | 1.03GB / 2.09GB [A[A
Processing Files (0 / 1) : 49%|████▉ | 1.03GB / 2.09GB, 67.9MB/s |
|
New Data Upload : 90%|█████████ | 1.03GB / 1.14GB, 67.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 50%|████▉ | 1.04GB / 2.09GB [A[A
Processing Files (0 / 1) : 50%|████▉ | 1.04GB / 2.09GB, 68.5MB/s |
|
New Data Upload : 91%|█████████▏| 1.04GB / 1.14GB, 68.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 51%|█████ | 1.06GB / 2.09GB [A[A
Processing Files (0 / 1) : 51%|█████ | 1.06GB / 2.09GB, 69.7MB/s |
|
New Data Upload : 93%|█████████▎| 1.06GB / 1.14GB, 69.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 52%|█████▏ | 1.08GB / 2.09GB [A[A
Processing Files (0 / 1) : 52%|█████▏ | 1.08GB / 2.09GB, 70.7MB/s |
|
New Data Upload : 90%|████████▉ | 1.08GB / 1.21GB, 70.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 53%|█████▎ | 1.11GB / 2.09GB [A[A
Processing Files (0 / 1) : 53%|█████▎ | 1.11GB / 2.09GB, 72.1MB/s |
|
New Data Upload : 92%|█████████▏| 1.11GB / 1.21GB, 72.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 54%|█████▍ | 1.12GB / 2.09GB [A[A
Processing Files (0 / 1) : 54%|█████▍ | 1.12GB / 2.09GB, 73.0MB/s |
|
New Data Upload : 93%|█████████▎| 1.12GB / 1.21GB, 73.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 55%|█████▍ | 1.15GB / 2.09GB [A[A
Processing Files (0 / 1) : 55%|█████▍ | 1.15GB / 2.09GB, 74.2MB/s |
|
New Data Upload : 95%|█████████▌| 1.15GB / 1.21GB, 74.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 56%|█████▌ | 1.16GB / 2.09GB [A[A
Processing Files (0 / 1) : 56%|█████▌ | 1.16GB / 2.09GB, 75.0MB/s |
|
New Data Upload : 91%|█████████▏| 1.16GB / 1.27GB, 75.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 56%|█████▋ | 1.17GB / 2.09GB [A[A
Processing Files (0 / 1) : 56%|█████▋ | 1.17GB / 2.09GB, 75.2MB/s |
|
New Data Upload : 92%|█████████▏| 1.17GB / 1.27GB, 75.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 57%|█████▋ | 1.19GB / 2.09GB [A[A
Processing Files (0 / 1) : 57%|█████▋ | 1.19GB / 2.09GB, 76.2MB/s |
|
New Data Upload : 94%|█████████▎| 1.19GB / 1.27GB, 76.2MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 58%|█████▊ | 1.21GB / 2.09GB [A[A
Processing Files (0 / 1) : 58%|█████▊ | 1.21GB / 2.09GB, 77.1MB/s |
|
New Data Upload : 90%|█████████ | 1.21GB / 1.34GB, 77.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 59%|█████▉ | 1.24GB / 2.09GB [A[A
Processing Files (0 / 1) : 59%|█████▉ | 1.24GB / 2.09GB, 78.4MB/s |
|
New Data Upload : 92%|█████████▏| 1.24GB / 1.34GB, 78.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 60%|█████▉ | 1.25GB / 2.09GB [A[A
Processing Files (0 / 1) : 60%|█████▉ | 1.25GB / 2.09GB, 78.6MB/s |
|
New Data Upload : 93%|█████████▎| 1.25GB / 1.34GB, 78.6MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 60%|██████ | 1.26GB / 2.09GB [A[A
Processing Files (0 / 1) : 60%|██████ | 1.26GB / 2.09GB, 79.1MB/s |
|
New Data Upload : 94%|█████████▍| 1.26GB / 1.34GB, 79.1MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 62%|██████▏ | 1.29GB / 2.09GB [A[A
Processing Files (0 / 1) : 62%|██████▏ | 1.29GB / 2.09GB, 80.5MB/s |
|
New Data Upload : 96%|█████████▌| 1.29GB / 1.34GB, 80.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 62%|██████▏ | 1.30GB / 2.09GB [A[A
Processing Files (0 / 1) : 62%|██████▏ | 1.30GB / 2.09GB, 81.0MB/s |
|
New Data Upload : 92%|█████████▏| 1.30GB / 1.41GB, 81.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 63%|██████▎ | 1.31GB / 2.09GB [A[A
Processing Files (0 / 1) : 63%|██████▎ | 1.31GB / 2.09GB, 81.5MB/s |
|
New Data Upload : 93%|█████████▎| 1.31GB / 1.41GB, 81.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 64%|██████▎ | 1.33GB / 2.09GB [A[A
Processing Files (0 / 1) : 64%|██████▎ | 1.33GB / 2.09GB, 82.0MB/s |
|
New Data Upload : 94%|█████████▍| 1.33GB / 1.41GB, 82.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 65%|██████▍ | 1.35GB / 2.09GB [A[A
Processing Files (0 / 1) : 65%|██████▍ | 1.35GB / 2.09GB, 83.0MB/s |
|
New Data Upload : 92%|█████████▏| 1.35GB / 1.48GB, 83.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 65%|██████▌ | 1.36GB / 2.09GB [A[A
Processing Files (0 / 1) : 65%|██████▌ | 1.36GB / 2.09GB, 83.4MB/s |
|
New Data Upload : 92%|█████████▏| 1.36GB / 1.48GB, 83.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 66%|██████▌ | 1.38GB / 2.09GB [A[A
Processing Files (0 / 1) : 66%|██████▌ | 1.38GB / 2.09GB, 83.7MB/s |
|
New Data Upload : 93%|█████████▎| 1.38GB / 1.48GB, 83.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 67%|██████▋ | 1.40GB / 2.09GB [A[A
Processing Files (0 / 1) : 67%|██████▋ | 1.40GB / 2.09GB, 84.7MB/s |
|
New Data Upload : 95%|█████████▍| 1.40GB / 1.48GB, 84.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 68%|██████▊ | 1.42GB / 2.09GB [A[A
Processing Files (0 / 1) : 68%|██████▊ | 1.42GB / 2.09GB, 85.7MB/s |
|
New Data Upload : 92%|█████████▏| 1.42GB / 1.54GB, 85.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 69%|██████▊ | 1.44GB / 2.09GB [A[A
Processing Files (0 / 1) : 69%|██████▊ | 1.44GB / 2.09GB, 86.4MB/s |
|
New Data Upload : 93%|█████████▎| 1.44GB / 1.54GB, 86.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 69%|██████▉ | 1.45GB / 2.09GB [A[A
Processing Files (0 / 1) : 69%|██████▉ | 1.45GB / 2.09GB, 86.5MB/s |
|
New Data Upload : 94%|█████████▎| 1.45GB / 1.54GB, 86.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 70%|███████ | 1.47GB / 2.09GB [A[A
Processing Files (0 / 1) : 70%|███████ | 1.47GB / 2.09GB, 87.4MB/s |
|
New Data Upload : 95%|█████████▌| 1.47GB / 1.54GB, 87.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 71%|███████ | 1.49GB / 2.09GB [A[A
Processing Files (0 / 1) : 71%|███████ | 1.49GB / 2.09GB, 88.3MB/s |
|
New Data Upload : 92%|█████████▏| 1.49GB / 1.61GB, 88.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 72%|███████▏ | 1.50GB / 2.09GB [A[A
Processing Files (0 / 1) : 72%|███████▏ | 1.50GB / 2.09GB, 88.7MB/s |
|
New Data Upload : 93%|█████████▎| 1.50GB / 1.61GB, 88.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 73%|███████▎ | 1.52GB / 2.09GB [A[A
Processing Files (0 / 1) : 73%|███████▎ | 1.52GB / 2.09GB, 89.3MB/s |
|
New Data Upload : 94%|█████████▍| 1.52GB / 1.61GB, 89.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 74%|███████▎ | 1.54GB / 2.09GB [A[A
Processing Files (0 / 1) : 74%|███████▎ | 1.54GB / 2.09GB, 90.3MB/s |
|
New Data Upload : 92%|█████████▏| 1.54GB / 1.68GB, 90.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 75%|███████▍ | 1.56GB / 2.09GB [A[A
Processing Files (0 / 1) : 75%|███████▍ | 1.56GB / 2.09GB, 91.5MB/s |
|
New Data Upload : 93%|█████████▎| 1.56GB / 1.68GB, 91.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 76%|███████▌ | 1.58GB / 2.09GB [A[A
Processing Files (0 / 1) : 76%|███████▌ | 1.58GB / 2.09GB, 92.4MB/s |
|
New Data Upload : 94%|█████████▍| 1.58GB / 1.68GB, 92.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 77%|███████▋ | 1.61GB / 2.09GB [A[A
Processing Files (0 / 1) : 77%|███████▋ | 1.61GB / 2.09GB, 93.5MB/s |
|
New Data Upload : 92%|█████████▏| 1.61GB / 1.74GB, 93.5MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 78%|███████▊ | 1.63GB / 2.09GB [A[A
Processing Files (0 / 1) : 78%|███████▊ | 1.63GB / 2.09GB, 94.4MB/s |
|
New Data Upload : 93%|█████████▎| 1.63GB / 1.74GB, 94.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 79%|███████▉ | 1.64GB / 2.09GB [A[A
Processing Files (0 / 1) : 79%|███████▉ | 1.64GB / 2.09GB, 95.0MB/s |
|
New Data Upload : 94%|█████████▍| 1.64GB / 1.74GB, 95.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 80%|███████▉ | 1.66GB / 2.09GB [A[A
Processing Files (0 / 1) : 80%|███████▉ | 1.66GB / 2.09GB, 95.7MB/s |
|
New Data Upload : 95%|█████████▌| 1.66GB / 1.74GB, 95.7MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 81%|████████ | 1.69GB / 2.09GB [A[A
Processing Files (0 / 1) : 81%|████████ | 1.69GB / 2.09GB, 96.8MB/s |
|
New Data Upload : 93%|█████████▎| 1.69GB / 1.81GB, 96.8MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 82%|████████▏ | 1.70GB / 2.09GB [A[A
Processing Files (0 / 1) : 82%|████████▏ | 1.70GB / 2.09GB, 97.4MB/s |
|
New Data Upload : 94%|█████████▍| 1.70GB / 1.81GB, 97.4MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 82%|████████▏ | 1.71GB / 2.09GB [A[A
Processing Files (0 / 1) : 82%|████████▏ | 1.71GB / 2.09GB, 97.3MB/s |
|
New Data Upload : 95%|█████████▍| 1.71GB / 1.81GB, 97.3MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 83%|████████▎ | 1.73GB / 2.09GB [A[A
Processing Files (0 / 1) : 83%|████████▎ | 1.73GB / 2.09GB, 98.0MB/s |
|
New Data Upload : 96%|█████████▌| 1.73GB / 1.81GB, 98.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 84%|████████▍ | 1.75GB / 2.09GB [A[A
Processing Files (0 / 1) : 84%|████████▍ | 1.75GB / 2.09GB, 99.0MB/s |
|
New Data Upload : 93%|█████████▎| 1.75GB / 1.88GB, 99.0MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 85%|████████▌ | 1.78GB / 2.09GB [A[A
Processing Files (0 / 1) : 85%|████████▌ | 1.78GB / 2.09GB, 99.9MB/s |
|
New Data Upload : 95%|█████████▍| 1.78GB / 1.88GB, 99.9MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 86%|████████▌ | 1.80GB / 2.09GB [A[A
Processing Files (0 / 1) : 86%|████████▌ | 1.80GB / 2.09GB, 101MB/s |
|
New Data Upload : 96%|█████████▌| 1.80GB / 1.88GB, 101MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 87%|████████▋ | 1.82GB / 2.09GB [A[A
Processing Files (0 / 1) : 87%|████████▋ | 1.82GB / 2.09GB, 102MB/s |
|
New Data Upload : 94%|█████████▎| 1.82GB / 1.94GB, 102MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 88%|████████▊ | 1.84GB / 2.09GB [A[A
Processing Files (0 / 1) : 88%|████████▊ | 1.84GB / 2.09GB, 103MB/s |
|
New Data Upload : 95%|█████████▍| 1.84GB / 1.94GB, 103MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 89%|████████▊ | 1.85GB / 2.09GB [A[A
Processing Files (0 / 1) : 89%|████████▊ | 1.85GB / 2.09GB, 103MB/s |
|
New Data Upload : 95%|█████████▌| 1.85GB / 1.94GB, 103MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 89%|████████▉ | 1.87GB / 2.09GB [A[A
Processing Files (0 / 1) : 89%|████████▉ | 1.87GB / 2.09GB, 103MB/s |
|
New Data Upload : 96%|█████████▌| 1.86GB / 1.94GB, 103MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 90%|█████████ | 1.88GB / 2.09GB [A[A
Processing Files (0 / 1) : 90%|█████████ | 1.88GB / 2.09GB, 104MB/s |
|
New Data Upload : 94%|█████████▎| 1.88GB / 2.01GB, 104MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 91%|█████████ | 1.90GB / 2.09GB [A[A
Processing Files (0 / 1) : 91%|█████████ | 1.90GB / 2.09GB, 104MB/s |
|
New Data Upload : 94%|█████████▍| 1.90GB / 2.01GB, 104MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 92%|█████████▏| 1.91GB / 2.09GB [A[A
Processing Files (0 / 1) : 92%|█████████▏| 1.91GB / 2.09GB, 104MB/s |
|
New Data Upload : 95%|█████████▌| 1.91GB / 2.01GB, 104MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 93%|█████████▎| 1.93GB / 2.09GB [A[A
Processing Files (0 / 1) : 93%|█████████▎| 1.93GB / 2.09GB, 105MB/s |
|
New Data Upload : 96%|█████████▌| 1.93GB / 2.01GB, 105MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 94%|█████████▎| 1.95GB / 2.09GB [A[A
Processing Files (0 / 1) : 94%|█████████▎| 1.95GB / 2.09GB, 106MB/s |
|
New Data Upload : 94%|█████████▍| 1.95GB / 2.08GB, 106MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 94%|█████████▍| 1.97GB / 2.09GB [A[A
Processing Files (0 / 1) : 94%|█████████▍| 1.97GB / 2.09GB, 106MB/s |
|
New Data Upload : 95%|█████████▍| 1.97GB / 2.08GB, 106MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 96%|█████████▌| 2.00GB / 2.09GB [A[A
Processing Files (0 / 1) : 96%|█████████▌| 2.00GB / 2.09GB, 107MB/s |
|
New Data Upload : 96%|█████████▌| 2.00GB / 2.09GB, 107MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 97%|█████████▋| 2.02GB / 2.09GB [A[A
Processing Files (0 / 1) : 97%|█████████▋| 2.02GB / 2.09GB, 108MB/s |
|
New Data Upload : 97%|█████████▋| 2.02GB / 2.09GB, 108MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 98%|█████████▊| 2.04GB / 2.09GB [A[A
Processing Files (0 / 1) : 98%|█████████▊| 2.04GB / 2.09GB, 109MB/s |
|
New Data Upload : 98%|█████████▊| 2.04GB / 2.09GB, 109MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 99%|█████████▊| 2.06GB / 2.09GB [A[A
Processing Files (0 / 1) : 99%|█████████▊| 2.06GB / 2.09GB, 110MB/s |
|
New Data Upload : 99%|█████████▊| 2.06GB / 2.09GB, 110MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 99%|█████████▉| 2.07GB / 2.09GB [A[A
Processing Files (0 / 1) : 99%|█████████▉| 2.07GB / 2.09GB, 110MB/s |
|
New Data Upload : 99%|█████████▉| 2.07GB / 2.09GB, 110MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.08GB / 2.09GB [A[A
Processing Files (0 / 1) : 100%|█████████▉| 2.08GB / 2.09GB, 110MB/s |
|
New Data Upload : 100%|█████████▉| 2.08GB / 2.09GB, 110MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A
Processing Files (0 / 1) : 100%|█████████▉| 2.09GB / 2.09GB, 109MB/s |
|
New Data Upload : 100%|█████████▉| 2.09GB / 2.09GB, 109MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB [A[A
Processing Files (0 / 1) : 100%|█████████▉| 2.09GB / 2.09GB, 105MB/s |
|
New Data Upload : 100%|█████████▉| 2.09GB / 2.09GB, 105MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A
Processing Files (1 / 1) : 100%|██████████| 2.09GB / 2.09GB, 104MB/s |
|
New Data Upload : 100%|██████████| 2.09GB / 2.09GB, 104MB/s [A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A |
|
|
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB [A[A
Processing Files (1 / 1) : 100%|██████████| 2.09GB / 2.09GB, 98.7MB/s |
|
New Data Upload : 100%|██████████| 2.09GB / 2.09GB, 98.6MB/s |
|
...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB |
| pushed |
| === ALL_DONE Sat Jun 20 15:24:50 UTC 2026 === |
| ==> Sat Jun 20 15:24:50 UTC 2026 SELF-DESTRUCT triggered (exit code=0) |
|
|
