==> Sat Jun 20 13:47:14 UTC 2026 start
NVIDIA A100-SXM4-80GB, 81920 MiB
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts hf, huggingface-cli and tiny-agents are installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
==> Sat Jun 20 13:47:25 UTC 2026 pull BCI from HF
BCI tar: /tmp/bci_data.tar
train pairs: 3896
X_1: torch.Size([3, 256, 256]) torch.float32 [-0.953, 0.718]
X_0: torch.Size([3, 256, 256]) torch.float32 [-0.129, 1.000]
==> Sat Jun 20 13:47:51 UTC 2026 TRAIN dual-task X_0 + rotation aug
[ddpm] T=1000  betas=(0.0001,0.02) alpha_bar[0]=0.999900 alpha_bar[-1]=4.035829e-05
[data] train=3896  HE/IHC paired, augment=True
[model] params=130.50M  in=6 out=3 (X_0-pred, DDPM)
[schedule] he_gen_prob=0.5 (0.5 = HistDiST 50/50, 0.0 = single-task baseline)
[iter 1/3000] loss=0.3243 L_he_ema=0.0000 L_ihc_ema=0.3243 frac_he=0.000 |conv_in grad: HE=0.00e+00 IHC=0.00e+00 ratio_I/H=0.00| wall=0.1min
[iter 100/3000] loss=0.0433 L_he_ema=0.0659 L_ihc_ema=0.0819 frac_he=0.535 |conv_in grad: HE=6.48e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=2.5min
[iter 200/3000] loss=0.0970 L_he_ema=0.0549 L_ihc_ema=0.0727 frac_he=0.580 |conv_in grad: HE=1.21e-01 IHC=7.29e-02 ratio_I/H=0.60| wall=5.0min
[iter 300/3000] loss=0.0266 L_he_ema=0.0557 L_ihc_ema=0.0435 frac_he=0.510 |conv_in grad: HE=5.90e-02 IHC=5.89e-02 ratio_I/H=1.00| wall=7.5min
[iter 400/3000] loss=0.0518 L_he_ema=0.0518 L_ihc_ema=0.0472 frac_he=0.400 |conv_in grad: HE=2.02e-01 IHC=1.89e-01 ratio_I/H=0.93| wall=10.0min
[iter 500/3000] loss=0.0473 L_he_ema=0.0524 L_ihc_ema=0.0533 frac_he=0.520 |conv_in grad: HE=5.30e-02 IHC=5.74e-02 ratio_I/H=1.08| wall=12.5min
[iter 600/3000] loss=0.0398 L_he_ema=0.0452 L_ihc_ema=0.0396 frac_he=0.530 |conv_in grad: HE=5.76e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=15.3min
[iter 700/3000] loss=0.0404 L_he_ema=0.0477 L_ihc_ema=0.0358 frac_he=0.510 |conv_in grad: HE=8.11e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=17.9min
[iter 800/3000] loss=0.0495 L_he_ema=0.0567 L_ihc_ema=0.0415 frac_he=0.480 |conv_in grad: HE=1.45e-01 IHC=0.00e+00 ratio_I/H=0.00| wall=20.6min
[iter 900/3000] loss=0.0306 L_he_ema=0.0458 L_ihc_ema=0.0321 frac_he=0.500 |conv_in grad: HE=1.99e-01 IHC=1.17e-01 ratio_I/H=0.59| wall=23.1min
[iter 1000/3000] loss=0.0425 L_he_ema=0.0618 L_ihc_ema=0.0466 frac_he=0.480 |conv_in grad: HE=6.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=25.6min
[save] iter 1000
[iter 1100/3000] loss=0.0342 L_he_ema=0.0417 L_ihc_ema=0.0385 frac_he=0.540 |conv_in grad: HE=1.91e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=28.5min
[iter 1200/3000] loss=0.0185 L_he_ema=0.0497 L_ihc_ema=0.0428 frac_he=0.550 |conv_in grad: HE=4.63e-02 IHC=4.81e-02 ratio_I/H=1.04| wall=31.3min
[iter 1300/3000] loss=0.0213 L_he_ema=0.0465 L_ihc_ema=0.0388 frac_he=0.570 |conv_in grad: HE=7.59e-02 IHC=3.32e-02 ratio_I/H=0.44| wall=33.9min
[iter 1400/3000] loss=0.0495 L_he_ema=0.0439 L_ihc_ema=0.0390 frac_he=0.540 |conv_in grad: HE=1.07e-01 IHC=0.00e+00 ratio_I/H=0.00| wall=36.4min
[iter 1500/3000] loss=0.0346 L_he_ema=0.0412 L_ihc_ema=0.0398 frac_he=0.580 |conv_in grad: HE=7.32e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=39.0min
[iter 1600/3000] loss=0.0277 L_he_ema=0.0380 L_ihc_ema=0.0272 frac_he=0.490 |conv_in grad: HE=9.41e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=41.7min
[iter 1700/3000] loss=0.0406 L_he_ema=0.0500 L_ihc_ema=0.0266 frac_he=0.460 |conv_in grad: HE=3.25e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=44.3min
[iter 1800/3000] loss=0.0179 L_he_ema=0.0426 L_ihc_ema=0.0313 frac_he=0.550 |conv_in grad: HE=2.29e-02 IHC=2.75e-02 ratio_I/H=1.20| wall=46.7min
[iter 1900/3000] loss=0.0278 L_he_ema=0.0395 L_ihc_ema=0.0303 frac_he=0.490 |conv_in grad: HE=4.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=49.2min
[iter 2000/3000] loss=0.0122 L_he_ema=0.0337 L_ihc_ema=0.0228 frac_he=0.560 |conv_in grad: HE=5.52e-02 IHC=3.47e-02 ratio_I/H=0.63| wall=51.8min
[save] iter 2000
[iter 2100/3000] loss=0.0365 L_he_ema=0.0459 L_ihc_ema=0.0349 frac_he=0.530 |conv_in grad: HE=4.97e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=54.7min
[iter 2200/3000] loss=0.0192 L_he_ema=0.0406 L_ihc_ema=0.0273 frac_he=0.520 |conv_in grad: HE=3.05e-02 IHC=3.19e-02 ratio_I/H=1.05| wall=57.2min
[iter 2300/3000] loss=0.0396 L_he_ema=0.0389 L_ihc_ema=0.0286 frac_he=0.440 |conv_in grad: HE=4.42e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=59.7min
[iter 2400/3000] loss=0.0231 L_he_ema=0.0355 L_ihc_ema=0.0253 frac_he=0.480 |conv_in grad: HE=8.65e-02 IHC=8.11e-02 ratio_I/H=0.94| wall=62.3min
[iter 2500/3000] loss=0.0450 L_he_ema=0.0381 L_ihc_ema=0.0259 frac_he=0.540 |conv_in grad: HE=6.44e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=64.8min
[iter 2600/3000] loss=0.0372 L_he_ema=0.0448 L_ihc_ema=0.0283 frac_he=0.450 |conv_in grad: HE=6.93e-02 IHC=6.99e-02 ratio_I/H=1.01| wall=67.6min
[iter 2700/3000] loss=0.0358 L_he_ema=0.0365 L_ihc_ema=0.0292 frac_he=0.510 |conv_in grad: HE=5.75e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=70.1min
[iter 2800/3000] loss=0.0494 L_he_ema=0.0458 L_ihc_ema=0.0303 frac_he=0.640 |conv_in grad: HE=1.86e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=72.7min
[iter 2900/3000] loss=0.0212 L_he_ema=0.0324 L_ihc_ema=0.0240 frac_he=0.480 |conv_in grad: HE=3.67e-02 IHC=0.00e+00 ratio_I/H=0.00| wall=75.2min
[iter 3000/3000] loss=0.0333 L_he_ema=0.0389 L_ihc_ema=0.0274 frac_he=0.460 |conv_in grad: HE=1.32e-01 IHC=1.54e-01 ratio_I/H=1.17| wall=77.8min
[save] iter 3000
[done] wall=78.1min
==> Sat Jun 20 15:06:15 UTC 2026 SAMPLE + FID NFE=100
[ckpt] iter=3000  he_gen_prob=0.5
[ema applied]
[data] n_test=977 batch=16
[sampled] 64/977
[sampled] 128/977
[sampled] 192/977
[sampled] 256/977
[sampled] 320/977
[sampled] 384/977
[sampled] 448/977
[sampled] 512/977
[sampled] 576/977
[sampled] 640/977
[sampled] 704/977
[sampled] 768/977
[sampled] 832/977
[sampled] 896/977
[sampled] 960/977
[sampled] 977/977
[done] 977 samples saved
compute FID between two folders
Found 977 images in the folder /home/ubuntu/work/dualtask_rot_fid/samples

FID samples :   0%|          | 0/16 [00:00<?, ?it/s]
FID samples :   6%|▋         | 1/16 [00:01<00:24,  1.64s/it]
FID samples :  12%|█▎        | 2/16 [00:06<00:46,  3.31s/it]
FID samples :  19%|█▉        | 3/16 [00:06<00:24,  1.91s/it]
FID samples :  25%|██▌       | 4/16 [00:06<00:14,  1.24s/it]
FID samples :  31%|███▏      | 5/16 [00:06<00:09,  1.15it/s]
FID samples :  38%|███▊      | 6/16 [00:07<00:06,  1.54it/s]
FID samples :  44%|████▍     | 7/16 [00:07<00:04,  1.96it/s]
FID samples :  50%|█████     | 8/16 [00:07<00:03,  2.42it/s]
FID samples :  56%|█████▋    | 9/16 [00:07<00:02,  2.79it/s]
FID samples :  62%|██████▎   | 10/16 [00:07<00:01,  3.01it/s]
FID samples :  69%|██████▉   | 11/16 [00:08<00:01,  3.14it/s]
FID samples :  75%|███████▌  | 12/16 [00:08<00:01,  3.32it/s]
FID samples :  81%|████████▏ | 13/16 [00:08<00:00,  3.61it/s]
FID samples :  88%|████████▊ | 14/16 [00:08<00:00,  3.67it/s]
FID samples :  94%|█████████▍| 15/16 [00:09<00:00,  3.77it/s]
FID samples : 100%|██████████| 16/16 [00:09<00:00,  3.53it/s]
FID samples : 100%|██████████| 16/16 [00:09<00:00,  1.65it/s]
Found 977 images in the folder /home/ubuntu/work/dualtask_rot_fid/target_ihc

FID target_ihc :   0%|          | 0/16 [00:00<?, ?it/s]
FID target_ihc :   6%|▋         | 1/16 [00:01<00:17,  1.18s/it]
FID target_ihc :  12%|█▎        | 2/16 [00:01<00:08,  1.59it/s]
FID target_ihc :  19%|█▉        | 3/16 [00:01<00:06,  1.92it/s]
FID target_ihc :  25%|██▌       | 4/16 [00:02<00:04,  2.45it/s]
FID target_ihc :  31%|███▏      | 5/16 [00:02<00:04,  2.54it/s]
FID target_ihc :  38%|███▊      | 6/16 [00:02<00:03,  2.97it/s]
FID target_ihc :  44%|████▍     | 7/16 [00:02<00:03,  2.94it/s]
FID target_ihc :  50%|█████     | 8/16 [00:03<00:02,  3.23it/s]
FID target_ihc :  56%|█████▋    | 9/16 [00:03<00:02,  3.06it/s]
FID target_ihc :  62%|██████▎   | 10/16 [00:03<00:01,  3.22it/s]
FID target_ihc :  69%|██████▉   | 11/16 [00:04<00:01,  3.28it/s]
FID target_ihc :  75%|███████▌  | 12/16 [00:04<00:01,  3.54it/s]
FID target_ihc :  81%|████████▏ | 13/16 [00:04<00:00,  3.16it/s]
FID target_ihc :  88%|████████▊ | 14/16 [00:05<00:00,  3.48it/s]
FID target_ihc :  94%|█████████▍| 15/16 [00:05<00:00,  3.26it/s]
FID target_ihc : 100%|██████████| 16/16 [00:08<00:00,  1.12s/it]
FID target_ihc : 100%|██████████| 16/16 [00:08<00:00,  1.87it/s]
FID (clean-fid, NFE=100) = 267.235
==> Sat Jun 20 15:24:24 UTC 2026 HF push dualtask

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            
Processing Files (0 / 1)      :   0%|          | 80.1kB / 2.09GB, 7.87kB/s  


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          | 80.1kB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   0%|          |  640kB / 2.09GB            
Processing Files (0 / 1)      :   0%|          |  640kB / 2.09GB, 61.4kB/s  

New Data Upload               :   1%|          |  560kB / 67.1MB, 54.0kB/s  


  ...task_rot/ckpt_iter3000.pt:   0%|          | 1.20MB / 2.09GB            
Processing Files (0 / 1)      :   0%|          | 1.20MB / 2.09GB,  115kB/s  

New Data Upload               :   1%|          | 1.12MB /  134MB,  107kB/s  


  ...task_rot/ckpt_iter3000.pt:   0%|          | 4.56MB / 2.09GB            
Processing Files (0 / 1)      :   0%|          | 4.56MB / 2.09GB,  436kB/s  

New Data Upload               :   3%|▎         | 4.48MB /  134MB,  429kB/s  


  ...task_rot/ckpt_iter3000.pt:   0%|          | 10.2MB / 2.09GB            
Processing Files (0 / 1)      :   0%|          | 10.2MB / 2.09GB,  968kB/s  

New Data Upload               :   8%|▊         | 10.1MB /  134MB,  960kB/s  


  ...task_rot/ckpt_iter3000.pt:   1%|          | 18.6MB / 2.09GB            
Processing Files (0 / 1)      :   1%|          | 18.6MB / 2.09GB, 1.76MB/s  

New Data Upload               :  14%|█▍        | 18.5MB /  134MB, 1.75MB/s  


  ...task_rot/ckpt_iter3000.pt:   1%|▏         | 30.9MB / 2.09GB            
Processing Files (0 / 1)      :   1%|▏         | 30.9MB / 2.09GB, 2.92MB/s  

New Data Upload               :  23%|██▎       | 30.8MB /  134MB, 2.91MB/s  


  ...task_rot/ckpt_iter3000.pt:   2%|▏         | 42.1MB / 2.09GB            
Processing Files (0 / 1)      :   2%|▏         | 42.1MB / 2.09GB, 3.95MB/s  

New Data Upload               :  31%|███▏      | 42.0MB /  134MB, 3.94MB/s  


  ...task_rot/ckpt_iter3000.pt:   3%|▎         | 53.8MB / 2.09GB            
Processing Files (0 / 1)      :   3%|▎         | 53.8MB / 2.09GB, 5.03MB/s  

New Data Upload               :  27%|██▋       | 53.7MB /  201MB, 5.02MB/s  


  ...task_rot/ckpt_iter3000.pt:   3%|▎         | 71.1MB / 2.09GB            
Processing Files (0 / 1)      :   3%|▎         | 71.1MB / 2.09GB, 6.62MB/s  

New Data Upload               :  35%|███▌      | 71.1MB /  201MB, 6.61MB/s  


  ...task_rot/ckpt_iter3000.pt:   4%|▍         | 89.6MB / 2.09GB            
Processing Files (0 / 1)      :   4%|▍         | 89.6MB / 2.09GB, 8.29MB/s  

New Data Upload               :  45%|████▍     | 89.5MB /  201MB, 8.28MB/s  


  ...task_rot/ckpt_iter3000.pt:   5%|▍         | 99.6MB / 2.09GB            
Processing Files (0 / 1)      :   5%|▍         | 99.6MB / 2.09GB, 9.16MB/s  

New Data Upload               :  50%|████▉     | 99.6MB /  201MB, 9.15MB/s  


  ...task_rot/ckpt_iter3000.pt:   5%|▌         |  110MB / 2.09GB            
Processing Files (0 / 1)      :   5%|▌         |  110MB / 2.09GB, 10.1MB/s  

New Data Upload               :  55%|█████▍    |  110MB /  201MB, 10.1MB/s  


  ...task_rot/ckpt_iter3000.pt:   6%|▌         |  121MB / 2.09GB            
Processing Files (0 / 1)      :   6%|▌         |  121MB / 2.09GB, 11.0MB/s  

New Data Upload               :  60%|██████    |  121MB /  201MB, 11.0MB/s  


  ...task_rot/ckpt_iter3000.pt:   6%|▌         |  130MB / 2.09GB            
Processing Files (0 / 1)      :   6%|▌         |  130MB / 2.09GB, 11.7MB/s  

New Data Upload               :  64%|██████▍   |  130MB /  201MB, 11.7MB/s  


  ...task_rot/ckpt_iter3000.pt:   6%|▋         |  134MB / 2.09GB            
Processing Files (0 / 1)      :   6%|▋         |  134MB / 2.09GB, 11.9MB/s  

New Data Upload               :  66%|██████▋   |  134MB /  201MB, 11.9MB/s  


  ...task_rot/ckpt_iter3000.pt:   6%|▋         |  134MB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   6%|▋         |  134MB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt:   6%|▋         |  134MB / 2.09GB            
Processing Files (0 / 1)      :   6%|▋         |  134MB / 2.09GB, 11.7MB/s  

New Data Upload               :  67%|██████▋   |  134MB /  201MB, 11.7MB/s  


  ...task_rot/ckpt_iter3000.pt:   7%|▋         |  139MB / 2.09GB            
Processing Files (0 / 1)      :   7%|▋         |  139MB / 2.09GB, 12.0MB/s  

New Data Upload               :  69%|██████▉   |  139MB /  201MB, 12.0MB/s  


  ...task_rot/ckpt_iter3000.pt:   7%|▋         |  150MB / 2.09GB            
Processing Files (0 / 1)      :   7%|▋         |  150MB / 2.09GB, 12.9MB/s  

New Data Upload               :  56%|█████▌    |  150MB /  268MB, 12.9MB/s  


  ...task_rot/ckpt_iter3000.pt:   8%|▊         |  159MB / 2.09GB            
Processing Files (0 / 1)      :   8%|▊         |  159MB / 2.09GB, 13.6MB/s  

New Data Upload               :  59%|█████▉    |  159MB /  268MB, 13.6MB/s  


  ...task_rot/ckpt_iter3000.pt:   8%|▊         |  168MB / 2.09GB            
Processing Files (0 / 1)      :   8%|▊         |  168MB / 2.09GB, 14.3MB/s  

New Data Upload               :  63%|██████▎   |  168MB /  268MB, 14.3MB/s  


  ...task_rot/ckpt_iter3000.pt:   8%|▊         |  175MB / 2.09GB            
Processing Files (0 / 1)      :   8%|▊         |  175MB / 2.09GB, 14.8MB/s  

New Data Upload               :  65%|██████▌   |  175MB /  268MB, 14.8MB/s  


  ...task_rot/ckpt_iter3000.pt:   9%|▉         |  196MB / 2.09GB            
Processing Files (0 / 1)      :   9%|▉         |  196MB / 2.09GB, 16.6MB/s  

New Data Upload               :  58%|█████▊    |  196MB /  335MB, 16.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  10%|█         |  216MB / 2.09GB            
Processing Files (0 / 1)      :  10%|█         |  216MB / 2.09GB, 18.2MB/s  

New Data Upload               :  64%|██████▍   |  216MB /  335MB, 18.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  11%|█         |  229MB / 2.09GB            
Processing Files (0 / 1)      :  11%|█         |  229MB / 2.09GB, 19.3MB/s  

New Data Upload               :  68%|██████▊   |  229MB /  335MB, 19.3MB/s  


  ...task_rot/ckpt_iter3000.pt:  12%|█▏        |  240MB / 2.09GB            
Processing Files (0 / 1)      :  12%|█▏        |  240MB / 2.09GB, 20.1MB/s  

New Data Upload               :  72%|███████▏  |  240MB /  335MB, 20.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  12%|█▏        |  250MB / 2.09GB            
Processing Files (0 / 1)      :  12%|█▏        |  250MB / 2.09GB, 20.8MB/s  

New Data Upload               :  75%|███████▍  |  250MB /  335MB, 20.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  12%|█▏        |  261MB / 2.09GB            
Processing Files (0 / 1)      :  12%|█▏        |  261MB / 2.09GB, 21.6MB/s  

New Data Upload               :  65%|██████▍   |  260MB /  402MB, 21.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  13%|█▎        |  272MB / 2.09GB            
Processing Files (0 / 1)      :  13%|█▎        |  272MB / 2.09GB, 22.4MB/s  

New Data Upload               :  68%|██████▊   |  272MB /  402MB, 22.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  13%|█▎        |  279MB / 2.09GB            
Processing Files (0 / 1)      :  13%|█▎        |  279MB / 2.09GB, 22.8MB/s  

New Data Upload               :  69%|██████▉   |  279MB /  402MB, 22.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  14%|█▍        |  290MB / 2.09GB            
Processing Files (0 / 1)      :  14%|█▍        |  290MB / 2.09GB, 23.6MB/s  

New Data Upload               :  72%|███████▏  |  290MB /  402MB, 23.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  14%|█▍        |  299MB / 2.09GB            
Processing Files (0 / 1)      :  14%|█▍        |  299MB / 2.09GB, 24.2MB/s  

New Data Upload               :  74%|███████▍  |  299MB /  402MB, 24.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  15%|█▍        |  311MB / 2.09GB            
Processing Files (0 / 1)      :  15%|█▍        |  311MB / 2.09GB, 25.1MB/s  

New Data Upload               :  66%|██████▋   |  311MB /  469MB, 25.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  16%|█▌        |  325MB / 2.09GB            
Processing Files (0 / 1)      :  16%|█▌        |  325MB / 2.09GB, 26.0MB/s  

New Data Upload               :  69%|██████▉   |  325MB /  469MB, 26.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  16%|█▋        |  340MB / 2.09GB            
Processing Files (0 / 1)      :  16%|█▋        |  340MB / 2.09GB, 27.2MB/s  

New Data Upload               :  73%|███████▎  |  340MB /  469MB, 27.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  18%|█▊        |  366MB / 2.09GB            
Processing Files (0 / 1)      :  18%|█▊        |  366MB / 2.09GB, 29.2MB/s  

New Data Upload               :  78%|███████▊  |  365MB /  469MB, 29.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  18%|█▊        |  383MB / 2.09GB            
Processing Files (0 / 1)      :  18%|█▊        |  383MB / 2.09GB, 30.5MB/s  

New Data Upload               :  82%|████████▏ |  383MB /  469MB, 30.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  20%|█▉        |  408MB / 2.09GB            
Processing Files (0 / 1)      :  20%|█▉        |  408MB / 2.09GB, 32.4MB/s  

New Data Upload               :  76%|███████▌  |  408MB /  536MB, 32.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  20%|██        |  427MB / 2.09GB            
Processing Files (0 / 1)      :  20%|██        |  427MB / 2.09GB, 33.7MB/s  

New Data Upload               :  80%|███████▉  |  427MB /  536MB, 33.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  21%|██▏       |  446MB / 2.09GB            
Processing Files (0 / 1)      :  21%|██▏       |  446MB / 2.09GB, 35.1MB/s  

New Data Upload               :  83%|████████▎ |  446MB /  536MB, 35.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  22%|██▏       |  456MB / 2.09GB            
Processing Files (0 / 1)      :  22%|██▏       |  456MB / 2.09GB, 35.7MB/s  

New Data Upload               :  85%|████████▌ |  456MB /  536MB, 35.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  23%|██▎       |  479MB / 2.09GB            
Processing Files (0 / 1)      :  23%|██▎       |  479MB / 2.09GB, 37.4MB/s  

New Data Upload               :  89%|████████▉ |  479MB /  536MB, 37.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  23%|██▎       |  488MB / 2.09GB            
Processing Files (0 / 1)      :  23%|██▎       |  488MB / 2.09GB, 37.9MB/s  

New Data Upload               :  81%|████████  |  488MB /  604MB, 37.9MB/s  


  ...task_rot/ckpt_iter3000.pt:  24%|██▍       |  500MB / 2.09GB            
Processing Files (0 / 1)      :  24%|██▍       |  500MB / 2.09GB, 38.6MB/s  

New Data Upload               :  83%|████████▎ |  500MB /  604MB, 38.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  24%|██▍       |  510MB / 2.09GB            
Processing Files (0 / 1)      :  24%|██▍       |  510MB / 2.09GB, 39.1MB/s  

New Data Upload               :  85%|████████▍ |  510MB /  604MB, 39.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  25%|██▌       |  526MB / 2.09GB            
Processing Files (0 / 1)      :  25%|██▌       |  526MB / 2.09GB, 40.1MB/s  

New Data Upload               :  87%|████████▋ |  525MB /  604MB, 40.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  26%|██▌       |  545MB / 2.09GB            
Processing Files (0 / 1)      :  26%|██▌       |  545MB / 2.09GB, 41.4MB/s  

New Data Upload               :  81%|████████▏ |  545MB /  671MB, 41.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  27%|██▋       |  560MB / 2.09GB            
Processing Files (0 / 1)      :  27%|██▋       |  560MB / 2.09GB, 42.4MB/s  

New Data Upload               :  84%|████████▎ |  560MB /  671MB, 42.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  27%|██▋       |  569MB / 2.09GB            
Processing Files (0 / 1)      :  27%|██▋       |  569MB / 2.09GB, 42.7MB/s  

New Data Upload               :  85%|████████▍ |  568MB /  671MB, 42.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  28%|██▊       |  577MB / 2.09GB            
Processing Files (0 / 1)      :  28%|██▊       |  577MB / 2.09GB, 43.1MB/s  

New Data Upload               :  86%|████████▌ |  577MB /  671MB, 43.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  28%|██▊       |  589MB / 2.09GB            
Processing Files (0 / 1)      :  28%|██▊       |  589MB / 2.09GB, 43.7MB/s  

New Data Upload               :  80%|███████▉  |  589MB /  738MB, 43.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  29%|██▊       |  599MB / 2.09GB            
Processing Files (0 / 1)      :  29%|██▊       |  599MB / 2.09GB, 44.2MB/s  

New Data Upload               :  81%|████████  |  599MB /  738MB, 44.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  29%|██▉       |  608MB / 2.09GB            
Processing Files (0 / 1)      :  29%|██▉       |  608MB / 2.09GB, 44.6MB/s  

New Data Upload               :  82%|████████▏ |  608MB /  738MB, 44.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  29%|██▉       |  615MB / 2.09GB            
Processing Files (0 / 1)      :  29%|██▉       |  615MB / 2.09GB, 44.8MB/s  

New Data Upload               :  83%|████████▎ |  614MB /  738MB, 44.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  30%|██▉       |  624MB / 2.09GB            
Processing Files (0 / 1)      :  30%|██▉       |  624MB / 2.09GB, 45.2MB/s  

New Data Upload               :  85%|████████▍ |  624MB /  738MB, 45.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  30%|███       |  636MB / 2.09GB            
Processing Files (0 / 1)      :  30%|███       |  636MB / 2.09GB, 45.9MB/s  

New Data Upload               :  79%|███████▉  |  636MB /  805MB, 45.9MB/s  


  ...task_rot/ckpt_iter3000.pt:  31%|███       |  648MB / 2.09GB            
Processing Files (0 / 1)      :  31%|███       |  648MB / 2.09GB, 46.5MB/s  

New Data Upload               :  81%|████████  |  648MB /  805MB, 46.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  32%|███▏      |  666MB / 2.09GB            
Processing Files (0 / 1)      :  32%|███▏      |  666MB / 2.09GB, 47.6MB/s  

New Data Upload               :  83%|████████▎ |  666MB /  805MB, 47.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  33%|███▎      |  689MB / 2.09GB            
Processing Files (0 / 1)      :  33%|███▎      |  689MB / 2.09GB, 49.2MB/s  

New Data Upload               :  86%|████████▌ |  689MB /  805MB, 49.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  35%|███▍      |  723MB / 2.09GB            
Processing Files (0 / 1)      :  35%|███▍      |  723MB / 2.09GB, 51.6MB/s  

New Data Upload               :  83%|████████▎ |  723MB /  872MB, 51.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  36%|███▌      |  747MB / 2.09GB            
Processing Files (0 / 1)      :  36%|███▌      |  747MB / 2.09GB, 53.2MB/s  

New Data Upload               :  86%|████████▌ |  747MB /  872MB, 53.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  37%|███▋      |  768MB / 2.09GB            
Processing Files (0 / 1)      :  37%|███▋      |  768MB / 2.09GB, 54.5MB/s  

New Data Upload               :  88%|████████▊ |  768MB /  872MB, 54.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  38%|███▊      |  789MB / 2.09GB            
Processing Files (0 / 1)      :  38%|███▊      |  789MB / 2.09GB, 55.8MB/s  

New Data Upload               :  90%|█████████ |  789MB /  872MB, 55.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  39%|███▊      |  809MB / 2.09GB            
Processing Files (0 / 1)      :  39%|███▊      |  809MB / 2.09GB, 57.0MB/s  

New Data Upload               :  86%|████████▌ |  809MB /  939MB, 57.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  40%|███▉      |  827MB / 2.09GB            
Processing Files (0 / 1)      :  40%|███▉      |  827MB / 2.09GB, 58.0MB/s  

New Data Upload               :  88%|████████▊ |  827MB /  939MB, 58.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  40%|████      |  837MB / 2.09GB            
Processing Files (0 / 1)      :  40%|████      |  837MB / 2.09GB, 58.4MB/s  

New Data Upload               :  89%|████████▉ |  837MB /  939MB, 58.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  40%|████      |  845MB / 2.09GB            
Processing Files (0 / 1)      :  40%|████      |  845MB / 2.09GB, 58.6MB/s  

New Data Upload               :  90%|█████████ |  845MB /  939MB, 58.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  42%|████▏     |  868MB / 2.09GB            
Processing Files (0 / 1)      :  42%|████▏     |  868MB / 2.09GB, 60.0MB/s  

New Data Upload               :  92%|█████████▏|  868MB /  939MB, 60.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  42%|████▏     |  883MB / 2.09GB            
Processing Files (0 / 1)      :  42%|████▏     |  883MB / 2.09GB, 60.8MB/s  

New Data Upload               :  88%|████████▊ |  883MB / 1.01GB, 60.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  43%|████▎     |  897MB / 2.09GB            
Processing Files (0 / 1)      :  43%|████▎     |  897MB / 2.09GB, 61.5MB/s  

New Data Upload               :  89%|████████▉ |  897MB / 1.01GB, 61.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  43%|████▎     |  908MB / 2.09GB            
Processing Files (0 / 1)      :  43%|████▎     |  908MB / 2.09GB, 61.8MB/s  

New Data Upload               :  90%|█████████ |  908MB / 1.01GB, 61.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  44%|████▍     |  921MB / 2.09GB            
Processing Files (0 / 1)      :  44%|████▍     |  921MB / 2.09GB, 62.4MB/s  

New Data Upload               :  92%|█████████▏|  921MB / 1.01GB, 62.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  45%|████▌     |  942MB / 2.09GB            
Processing Files (0 / 1)      :  45%|████▌     |  942MB / 2.09GB, 63.6MB/s  

New Data Upload               :  88%|████████▊ |  942MB / 1.07GB, 63.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  46%|████▌     |  963MB / 2.09GB            
Processing Files (0 / 1)      :  46%|████▌     |  963MB / 2.09GB, 64.9MB/s  

New Data Upload               :  90%|████████▉ |  963MB / 1.07GB, 64.9MB/s  


  ...task_rot/ckpt_iter3000.pt:  47%|████▋     |  975MB / 2.09GB            
Processing Files (0 / 1)      :  47%|████▋     |  975MB / 2.09GB, 65.2MB/s  

New Data Upload               :  91%|█████████ |  975MB / 1.07GB, 65.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  47%|████▋     |  986MB / 2.09GB            
Processing Files (0 / 1)      :  47%|████▋     |  986MB / 2.09GB, 65.6MB/s  

New Data Upload               :  92%|█████████▏|  986MB / 1.07GB, 65.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  48%|████▊     | 1.01GB / 2.09GB            
Processing Files (0 / 1)      :  48%|████▊     | 1.01GB / 2.09GB, 67.1MB/s  

New Data Upload               :  89%|████████▊ | 1.01GB / 1.14GB, 67.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  49%|████▉     | 1.03GB / 2.09GB            
Processing Files (0 / 1)      :  49%|████▉     | 1.03GB / 2.09GB, 67.9MB/s  

New Data Upload               :  90%|█████████ | 1.03GB / 1.14GB, 67.9MB/s  


  ...task_rot/ckpt_iter3000.pt:  50%|████▉     | 1.04GB / 2.09GB            
Processing Files (0 / 1)      :  50%|████▉     | 1.04GB / 2.09GB, 68.5MB/s  

New Data Upload               :  91%|█████████▏| 1.04GB / 1.14GB, 68.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  51%|█████     | 1.06GB / 2.09GB            
Processing Files (0 / 1)      :  51%|█████     | 1.06GB / 2.09GB, 69.7MB/s  

New Data Upload               :  93%|█████████▎| 1.06GB / 1.14GB, 69.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  52%|█████▏    | 1.08GB / 2.09GB            
Processing Files (0 / 1)      :  52%|█████▏    | 1.08GB / 2.09GB, 70.7MB/s  

New Data Upload               :  90%|████████▉ | 1.08GB / 1.21GB, 70.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  53%|█████▎    | 1.11GB / 2.09GB            
Processing Files (0 / 1)      :  53%|█████▎    | 1.11GB / 2.09GB, 72.1MB/s  

New Data Upload               :  92%|█████████▏| 1.11GB / 1.21GB, 72.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  54%|█████▍    | 1.12GB / 2.09GB            
Processing Files (0 / 1)      :  54%|█████▍    | 1.12GB / 2.09GB, 73.0MB/s  

New Data Upload               :  93%|█████████▎| 1.12GB / 1.21GB, 73.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  55%|█████▍    | 1.15GB / 2.09GB            
Processing Files (0 / 1)      :  55%|█████▍    | 1.15GB / 2.09GB, 74.2MB/s  

New Data Upload               :  95%|█████████▌| 1.15GB / 1.21GB, 74.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  56%|█████▌    | 1.16GB / 2.09GB            
Processing Files (0 / 1)      :  56%|█████▌    | 1.16GB / 2.09GB, 75.0MB/s  

New Data Upload               :  91%|█████████▏| 1.16GB / 1.27GB, 75.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  56%|█████▋    | 1.17GB / 2.09GB            
Processing Files (0 / 1)      :  56%|█████▋    | 1.17GB / 2.09GB, 75.2MB/s  

New Data Upload               :  92%|█████████▏| 1.17GB / 1.27GB, 75.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  57%|█████▋    | 1.19GB / 2.09GB            
Processing Files (0 / 1)      :  57%|█████▋    | 1.19GB / 2.09GB, 76.2MB/s  

New Data Upload               :  94%|█████████▎| 1.19GB / 1.27GB, 76.2MB/s  


  ...task_rot/ckpt_iter3000.pt:  58%|█████▊    | 1.21GB / 2.09GB            
Processing Files (0 / 1)      :  58%|█████▊    | 1.21GB / 2.09GB, 77.1MB/s  

New Data Upload               :  90%|█████████ | 1.21GB / 1.34GB, 77.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  59%|█████▉    | 1.24GB / 2.09GB            
Processing Files (0 / 1)      :  59%|█████▉    | 1.24GB / 2.09GB, 78.4MB/s  

New Data Upload               :  92%|█████████▏| 1.24GB / 1.34GB, 78.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  60%|█████▉    | 1.25GB / 2.09GB            
Processing Files (0 / 1)      :  60%|█████▉    | 1.25GB / 2.09GB, 78.6MB/s  

New Data Upload               :  93%|█████████▎| 1.25GB / 1.34GB, 78.6MB/s  


  ...task_rot/ckpt_iter3000.pt:  60%|██████    | 1.26GB / 2.09GB            
Processing Files (0 / 1)      :  60%|██████    | 1.26GB / 2.09GB, 79.1MB/s  

New Data Upload               :  94%|█████████▍| 1.26GB / 1.34GB, 79.1MB/s  


  ...task_rot/ckpt_iter3000.pt:  62%|██████▏   | 1.29GB / 2.09GB            
Processing Files (0 / 1)      :  62%|██████▏   | 1.29GB / 2.09GB, 80.5MB/s  

New Data Upload               :  96%|█████████▌| 1.29GB / 1.34GB, 80.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  62%|██████▏   | 1.30GB / 2.09GB            
Processing Files (0 / 1)      :  62%|██████▏   | 1.30GB / 2.09GB, 81.0MB/s  

New Data Upload               :  92%|█████████▏| 1.30GB / 1.41GB, 81.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  63%|██████▎   | 1.31GB / 2.09GB            
Processing Files (0 / 1)      :  63%|██████▎   | 1.31GB / 2.09GB, 81.5MB/s  

New Data Upload               :  93%|█████████▎| 1.31GB / 1.41GB, 81.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  64%|██████▎   | 1.33GB / 2.09GB            
Processing Files (0 / 1)      :  64%|██████▎   | 1.33GB / 2.09GB, 82.0MB/s  

New Data Upload               :  94%|█████████▍| 1.33GB / 1.41GB, 82.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  65%|██████▍   | 1.35GB / 2.09GB            
Processing Files (0 / 1)      :  65%|██████▍   | 1.35GB / 2.09GB, 83.0MB/s  

New Data Upload               :  92%|█████████▏| 1.35GB / 1.48GB, 83.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  65%|██████▌   | 1.36GB / 2.09GB            
Processing Files (0 / 1)      :  65%|██████▌   | 1.36GB / 2.09GB, 83.4MB/s  

New Data Upload               :  92%|█████████▏| 1.36GB / 1.48GB, 83.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  66%|██████▌   | 1.38GB / 2.09GB            
Processing Files (0 / 1)      :  66%|██████▌   | 1.38GB / 2.09GB, 83.7MB/s  

New Data Upload               :  93%|█████████▎| 1.38GB / 1.48GB, 83.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  67%|██████▋   | 1.40GB / 2.09GB            
Processing Files (0 / 1)      :  67%|██████▋   | 1.40GB / 2.09GB, 84.7MB/s  

New Data Upload               :  95%|█████████▍| 1.40GB / 1.48GB, 84.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  68%|██████▊   | 1.42GB / 2.09GB            
Processing Files (0 / 1)      :  68%|██████▊   | 1.42GB / 2.09GB, 85.7MB/s  

New Data Upload               :  92%|█████████▏| 1.42GB / 1.54GB, 85.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  69%|██████▊   | 1.44GB / 2.09GB            
Processing Files (0 / 1)      :  69%|██████▊   | 1.44GB / 2.09GB, 86.4MB/s  

New Data Upload               :  93%|█████████▎| 1.44GB / 1.54GB, 86.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  69%|██████▉   | 1.45GB / 2.09GB            
Processing Files (0 / 1)      :  69%|██████▉   | 1.45GB / 2.09GB, 86.5MB/s  

New Data Upload               :  94%|█████████▎| 1.45GB / 1.54GB, 86.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  70%|███████   | 1.47GB / 2.09GB            
Processing Files (0 / 1)      :  70%|███████   | 1.47GB / 2.09GB, 87.4MB/s  

New Data Upload               :  95%|█████████▌| 1.47GB / 1.54GB, 87.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  71%|███████   | 1.49GB / 2.09GB            
Processing Files (0 / 1)      :  71%|███████   | 1.49GB / 2.09GB, 88.3MB/s  

New Data Upload               :  92%|█████████▏| 1.49GB / 1.61GB, 88.3MB/s  


  ...task_rot/ckpt_iter3000.pt:  72%|███████▏  | 1.50GB / 2.09GB            
Processing Files (0 / 1)      :  72%|███████▏  | 1.50GB / 2.09GB, 88.7MB/s  

New Data Upload               :  93%|█████████▎| 1.50GB / 1.61GB, 88.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  73%|███████▎  | 1.52GB / 2.09GB            
Processing Files (0 / 1)      :  73%|███████▎  | 1.52GB / 2.09GB, 89.3MB/s  

New Data Upload               :  94%|█████████▍| 1.52GB / 1.61GB, 89.3MB/s  


  ...task_rot/ckpt_iter3000.pt:  74%|███████▎  | 1.54GB / 2.09GB            
Processing Files (0 / 1)      :  74%|███████▎  | 1.54GB / 2.09GB, 90.3MB/s  

New Data Upload               :  92%|█████████▏| 1.54GB / 1.68GB, 90.3MB/s  


  ...task_rot/ckpt_iter3000.pt:  75%|███████▍  | 1.56GB / 2.09GB            
Processing Files (0 / 1)      :  75%|███████▍  | 1.56GB / 2.09GB, 91.5MB/s  

New Data Upload               :  93%|█████████▎| 1.56GB / 1.68GB, 91.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  76%|███████▌  | 1.58GB / 2.09GB            
Processing Files (0 / 1)      :  76%|███████▌  | 1.58GB / 2.09GB, 92.4MB/s  

New Data Upload               :  94%|█████████▍| 1.58GB / 1.68GB, 92.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  77%|███████▋  | 1.61GB / 2.09GB            
Processing Files (0 / 1)      :  77%|███████▋  | 1.61GB / 2.09GB, 93.5MB/s  

New Data Upload               :  92%|█████████▏| 1.61GB / 1.74GB, 93.5MB/s  


  ...task_rot/ckpt_iter3000.pt:  78%|███████▊  | 1.63GB / 2.09GB            
Processing Files (0 / 1)      :  78%|███████▊  | 1.63GB / 2.09GB, 94.4MB/s  

New Data Upload               :  93%|█████████▎| 1.63GB / 1.74GB, 94.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  79%|███████▉  | 1.64GB / 2.09GB            
Processing Files (0 / 1)      :  79%|███████▉  | 1.64GB / 2.09GB, 95.0MB/s  

New Data Upload               :  94%|█████████▍| 1.64GB / 1.74GB, 95.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  80%|███████▉  | 1.66GB / 2.09GB            
Processing Files (0 / 1)      :  80%|███████▉  | 1.66GB / 2.09GB, 95.7MB/s  

New Data Upload               :  95%|█████████▌| 1.66GB / 1.74GB, 95.7MB/s  


  ...task_rot/ckpt_iter3000.pt:  81%|████████  | 1.69GB / 2.09GB            
Processing Files (0 / 1)      :  81%|████████  | 1.69GB / 2.09GB, 96.8MB/s  

New Data Upload               :  93%|█████████▎| 1.69GB / 1.81GB, 96.8MB/s  


  ...task_rot/ckpt_iter3000.pt:  82%|████████▏ | 1.70GB / 2.09GB            
Processing Files (0 / 1)      :  82%|████████▏ | 1.70GB / 2.09GB, 97.4MB/s  

New Data Upload               :  94%|█████████▍| 1.70GB / 1.81GB, 97.4MB/s  


  ...task_rot/ckpt_iter3000.pt:  82%|████████▏ | 1.71GB / 2.09GB            
Processing Files (0 / 1)      :  82%|████████▏ | 1.71GB / 2.09GB, 97.3MB/s  

New Data Upload               :  95%|█████████▍| 1.71GB / 1.81GB, 97.3MB/s  


  ...task_rot/ckpt_iter3000.pt:  83%|████████▎ | 1.73GB / 2.09GB            
Processing Files (0 / 1)      :  83%|████████▎ | 1.73GB / 2.09GB, 98.0MB/s  

New Data Upload               :  96%|█████████▌| 1.73GB / 1.81GB, 98.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  84%|████████▍ | 1.75GB / 2.09GB            
Processing Files (0 / 1)      :  84%|████████▍ | 1.75GB / 2.09GB, 99.0MB/s  

New Data Upload               :  93%|█████████▎| 1.75GB / 1.88GB, 99.0MB/s  


  ...task_rot/ckpt_iter3000.pt:  85%|████████▌ | 1.78GB / 2.09GB            
Processing Files (0 / 1)      :  85%|████████▌ | 1.78GB / 2.09GB, 99.9MB/s  

New Data Upload               :  95%|█████████▍| 1.78GB / 1.88GB, 99.9MB/s  


  ...task_rot/ckpt_iter3000.pt:  86%|████████▌ | 1.80GB / 2.09GB            
Processing Files (0 / 1)      :  86%|████████▌ | 1.80GB / 2.09GB,  101MB/s  

New Data Upload               :  96%|█████████▌| 1.80GB / 1.88GB,  101MB/s  


  ...task_rot/ckpt_iter3000.pt:  87%|████████▋ | 1.82GB / 2.09GB            
Processing Files (0 / 1)      :  87%|████████▋ | 1.82GB / 2.09GB,  102MB/s  

New Data Upload               :  94%|█████████▎| 1.82GB / 1.94GB,  102MB/s  


  ...task_rot/ckpt_iter3000.pt:  88%|████████▊ | 1.84GB / 2.09GB            
Processing Files (0 / 1)      :  88%|████████▊ | 1.84GB / 2.09GB,  103MB/s  

New Data Upload               :  95%|█████████▍| 1.84GB / 1.94GB,  103MB/s  


  ...task_rot/ckpt_iter3000.pt:  89%|████████▊ | 1.85GB / 2.09GB            
Processing Files (0 / 1)      :  89%|████████▊ | 1.85GB / 2.09GB,  103MB/s  

New Data Upload               :  95%|█████████▌| 1.85GB / 1.94GB,  103MB/s  


  ...task_rot/ckpt_iter3000.pt:  89%|████████▉ | 1.87GB / 2.09GB            
Processing Files (0 / 1)      :  89%|████████▉ | 1.87GB / 2.09GB,  103MB/s  

New Data Upload               :  96%|█████████▌| 1.86GB / 1.94GB,  103MB/s  


  ...task_rot/ckpt_iter3000.pt:  90%|█████████ | 1.88GB / 2.09GB            
Processing Files (0 / 1)      :  90%|█████████ | 1.88GB / 2.09GB,  104MB/s  

New Data Upload               :  94%|█████████▎| 1.88GB / 2.01GB,  104MB/s  


  ...task_rot/ckpt_iter3000.pt:  91%|█████████ | 1.90GB / 2.09GB            
Processing Files (0 / 1)      :  91%|█████████ | 1.90GB / 2.09GB,  104MB/s  

New Data Upload               :  94%|█████████▍| 1.90GB / 2.01GB,  104MB/s  


  ...task_rot/ckpt_iter3000.pt:  92%|█████████▏| 1.91GB / 2.09GB            
Processing Files (0 / 1)      :  92%|█████████▏| 1.91GB / 2.09GB,  104MB/s  

New Data Upload               :  95%|█████████▌| 1.91GB / 2.01GB,  104MB/s  


  ...task_rot/ckpt_iter3000.pt:  93%|█████████▎| 1.93GB / 2.09GB            
Processing Files (0 / 1)      :  93%|█████████▎| 1.93GB / 2.09GB,  105MB/s  

New Data Upload               :  96%|█████████▌| 1.93GB / 2.01GB,  105MB/s  


  ...task_rot/ckpt_iter3000.pt:  94%|█████████▎| 1.95GB / 2.09GB            
Processing Files (0 / 1)      :  94%|█████████▎| 1.95GB / 2.09GB,  106MB/s  

New Data Upload               :  94%|█████████▍| 1.95GB / 2.08GB,  106MB/s  


  ...task_rot/ckpt_iter3000.pt:  94%|█████████▍| 1.97GB / 2.09GB            
Processing Files (0 / 1)      :  94%|█████████▍| 1.97GB / 2.09GB,  106MB/s  

New Data Upload               :  95%|█████████▍| 1.97GB / 2.08GB,  106MB/s  


  ...task_rot/ckpt_iter3000.pt:  96%|█████████▌| 2.00GB / 2.09GB            
Processing Files (0 / 1)      :  96%|█████████▌| 2.00GB / 2.09GB,  107MB/s  

New Data Upload               :  96%|█████████▌| 2.00GB / 2.09GB,  107MB/s  


  ...task_rot/ckpt_iter3000.pt:  97%|█████████▋| 2.02GB / 2.09GB            
Processing Files (0 / 1)      :  97%|█████████▋| 2.02GB / 2.09GB,  108MB/s  

New Data Upload               :  97%|█████████▋| 2.02GB / 2.09GB,  108MB/s  


  ...task_rot/ckpt_iter3000.pt:  98%|█████████▊| 2.04GB / 2.09GB            
Processing Files (0 / 1)      :  98%|█████████▊| 2.04GB / 2.09GB,  109MB/s  

New Data Upload               :  98%|█████████▊| 2.04GB / 2.09GB,  109MB/s  


  ...task_rot/ckpt_iter3000.pt:  99%|█████████▊| 2.06GB / 2.09GB            
Processing Files (0 / 1)      :  99%|█████████▊| 2.06GB / 2.09GB,  110MB/s  

New Data Upload               :  99%|█████████▊| 2.06GB / 2.09GB,  110MB/s  


  ...task_rot/ckpt_iter3000.pt:  99%|█████████▉| 2.07GB / 2.09GB            
Processing Files (0 / 1)      :  99%|█████████▉| 2.07GB / 2.09GB,  110MB/s  

New Data Upload               :  99%|█████████▉| 2.07GB / 2.09GB,  110MB/s  


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.08GB / 2.09GB            
Processing Files (0 / 1)      : 100%|█████████▉| 2.08GB / 2.09GB,  110MB/s  

New Data Upload               : 100%|█████████▉| 2.08GB / 2.09GB,  110MB/s  


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            
Processing Files (0 / 1)      : 100%|█████████▉| 2.09GB / 2.09GB,  109MB/s  

New Data Upload               : 100%|█████████▉| 2.09GB / 2.09GB,  109MB/s  


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|█████████▉| 2.09GB / 2.09GB            
Processing Files (0 / 1)      : 100%|█████████▉| 2.09GB / 2.09GB,  105MB/s  

New Data Upload               : 100%|█████████▉| 2.09GB / 2.09GB,  105MB/s  


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            
Processing Files (1 / 1)      : 100%|██████████| 2.09GB / 2.09GB,  104MB/s  

New Data Upload               : 100%|██████████| 2.09GB / 2.09GB,  104MB/s  


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            


  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            
Processing Files (1 / 1)      : 100%|██████████| 2.09GB / 2.09GB, 98.7MB/s  

New Data Upload               : 100%|██████████| 2.09GB / 2.09GB, 98.6MB/s  

  ...task_rot/ckpt_iter3000.pt: 100%|██████████| 2.09GB / 2.09GB            
pushed
=== ALL_DONE Sat Jun 20 15:24:50 UTC 2026 ===
==> Sat Jun 20 15:24:50 UTC 2026 SELF-DESTRUCT triggered (exit code=0)