W0530 22:18:25.521000 714796 torch/distributed/run.py:862] W0530 22:18:25.521000 714796 torch/distributed/run.py:862] ***************************************** W0530 22:18:25.521000 714796 torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0530 22:18:25.521000 714796 torch/distributed/run.py:862] ***************************************** Config: {'project': 'thoughtformer', 'name': 'star-datacurr-only-4gpu-bs32-lr1e-5', 'only_eval': False, 'method': 'thoughtformer', 'data_curriculum': True, 'thought_curriculum': False, 'staging': 'adaptive', 'init_thought_stage': 1, 'init_data_stage': 1, 'patience': 5, 'staging_threshold': 0.9, 'c_thought': 1, 'max_latent_stage': 10, 'save_only_improve': False, 'model_id': 'Qwen/Qwen3-0.6B-Base', 'load_model_path': 'None', 'seed': 0, 'resume': 0, 'bf16': False, 'train_path': 'data/star_k10_L10_1000_train.json', 'val_path': 'data/star_k10_L10_100_valid.json', 'reset_optimizer': False, 'lr_schedule': 'none', 'batch_size_training': 4, 'eval_only_trained': False, 'eval_batch_size': 8, 'eval_every': 1, 'debug': False, 'gradient_accumulation_steps': 2, 'num_epochs': 100, 'lr': 1e-05, 'weight_decay': 0.01, 'group': 'star_datacurr_only', 'tags': ['star', 'k10', 'L10', 'data_curr_only', 'fixed_thought_10', 'fixed_lr', 'no_uniform'], 'run_type': 'pilot'} /home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [rank0]:[W530 22:18:31.109903417 ProcessGroupNCCL.cpp:5324] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() Loading weights: 0%| | 0/310 [00:00= 0.9 *** *** Advancing to stage 2 *** Warning: failed to upload checkpoint_4 to HF: (Request ID: Root=1-6a1b68fe-5ef298115eee74b433674140;1168b31d-8889-4b0c-af0d-2219279dadab) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 4/100 (thought_stage=10, data_stage=2) ============================================================ thought_stage=10, c_thought=1, max_difficulty=2 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 3 *** Warning: failed to upload checkpoint_6 to HF: (Request ID: Root=1-6a1b6dd2-12fe65891c22ba5a6e89a8df;0124666f-2597-456a-b4f7-a8f2bc9c1025) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 6/100 (thought_stage=10, data_stage=3) ============================================================ thought_stage=10, c_thought=1, max_difficulty=3 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 4 *** Warning: failed to upload checkpoint_8 to HF: (Request ID: Root=1-6a1b7435-3c6484f25acda7746a774627;cee7dfce-1dd7-453b-be6d-d03af3214ad2) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 8/100 (thought_stage=10, data_stage=4) ============================================================ thought_stage=10, c_thought=1, max_difficulty=4 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 5 *** Warning: failed to upload checkpoint_10 to HF: (Request ID: Root=1-6a1b7c18-5635416a6c432b5f2b011e95;cb37aabe-dc31-4f1d-86a5-959703488ccb) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 10/100 (thought_stage=10, data_stage=5) ============================================================ thought_stage=10, c_thought=1, max_difficulty=5 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 6 *** Warning: failed to upload checkpoint_11 to HF: (Request ID: Root=1-6a1b80cc-6eb663257bba71cc351fed23;6a30aa55-11f0-48db-85e1-1306f9c780ce) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 11/100 (thought_stage=10, data_stage=6) ============================================================ thought_stage=10, c_thought=1, max_difficulty=6 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 7 *** Warning: failed to upload checkpoint_13 to HF: (Request ID: Root=1-6a1b8bba-60022d5014cd386e705139ce;739ae0e8-f85d-41b4-a9e0-de435c8b8ca8) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 13/100 (thought_stage=10, data_stage=7) ============================================================ thought_stage=10, c_thought=1, max_difficulty=7 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 8 *** Processing Files (0 / 0) : | | 0.00B / 0.00B New Data Upload : | | 0.00B / 0.00B  ...ve-03b89ee9/checkpoint_14: 10%|▉ | 360MB / 3.63GB  ...ve-03b89ee9/checkpoint_14: 10%|▉ | 360MB / 3.63GB  Processing Files (0 / 1) : 10%|▉ | 360MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_14: 20%|██ | 736MB / 3.63GB  Processing Files (0 / 1) : 20%|██ | 736MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_14: 31%|███ | 1.11GB / 3.63GB  Processing Files (0 / 1) : 31%|███ | 1.11GB / 3.63GB, 107MB/s ...ve-03b89ee9/checkpoint_14: 41%|████ | 1.49GB / 3.63GB  Processing Files (0 / 1) : 41%|████ | 1.49GB / 3.63GB, 141MB/s ...ve-03b89ee9/checkpoint_14: 51%|█████ | 1.86GB / 3.63GB  Processing Files (0 / 1) : 51%|█████ | 1.86GB / 3.63GB, 173MB/s ...ve-03b89ee9/checkpoint_14: 61%|██████▏ | 2.22GB / 3.63GB  Processing Files (0 / 1) : 61%|██████▏ | 2.22GB / 3.63GB, 204MB/s ...ve-03b89ee9/checkpoint_14: 72%|███████▏ | 2.60GB / 3.63GB  Processing Files (0 / 1) : 72%|███████▏ | 2.60GB / 3.63GB, 235MB/s ...ve-03b89ee9/checkpoint_14: 82%|████████▏ | 2.97GB / 3.63GB  Processing Files (0 / 1) : 82%|████████▏ | 2.97GB / 3.63GB, 264MB/s ...ve-03b89ee9/checkpoint_14: 92%|█████████▏| 3.34GB / 3.63GB  Processing Files (0 / 1) : 92%|█████████▏| 3.34GB / 3.63GB, 293MB/s ...ve-03b89ee9/checkpoint_14: 100%|██████████| 3.63GB / 3.63GB  Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s New Data Upload : | | 0.00B / 0.00B, 0.00B/s ...ve-03b89ee9/checkpoint_14: 100%|██████████| 3.63GB / 3.63GB No files have been modified since last commit. Skipping to prevent empty commit. Uploaded checkpoint_14 to seyedparsa/thoughtformer-checkpoints/checkpoints/star-datacurr-only-4gpu-bs32-lr1e-5_n1000_qwen3-0.6b-base_thoughtformer_lr1e-5_no-reset_dc_no-tc_adaptive-03b89ee9/checkpoint_14 ============================================================ EPOCH 14/100 (thought_stage=10, data_stage=8) ============================================================ thought_stage=10, c_thought=1, max_difficulty=8 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 9 *** Processing Files (0 / 0) : | | 0.00B / 0.00B New Data Upload : | | 0.00B / 0.00B  ...ve-03b89ee9/checkpoint_15: 10%|▉ | 352MB / 3.63GB  ...ve-03b89ee9/checkpoint_15: 10%|▉ | 352MB / 3.63GB  Processing Files (0 / 1) : 10%|▉ | 352MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_15: 20%|██ | 728MB / 3.63GB  Processing Files (0 / 1) : 20%|██ | 728MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_15: 30%|███ | 1.10GB / 3.63GB  Processing Files (0 / 1) : 30%|███ | 1.10GB / 3.63GB, 106MB/s ...ve-03b89ee9/checkpoint_15: 41%|████ | 1.47GB / 3.63GB  Processing Files (0 / 1) : 41%|████ | 1.47GB / 3.63GB, 139MB/s ...ve-03b89ee9/checkpoint_15: 51%|█████ | 1.86GB / 3.63GB  Processing Files (0 / 1) : 51%|█████ | 1.86GB / 3.63GB, 173MB/s ...ve-03b89ee9/checkpoint_15: 62%|██████▏ | 2.23GB / 3.63GB  Processing Files (0 / 1) : 62%|██████▏ | 2.23GB / 3.63GB, 205MB/s ...ve-03b89ee9/checkpoint_15: 72%|███████▏ | 2.61GB / 3.63GB  Processing Files (0 / 1) : 72%|███████▏ | 2.61GB / 3.63GB, 236MB/s ...ve-03b89ee9/checkpoint_15: 82%|████████▏ | 2.98GB / 3.63GB  Processing Files (0 / 1) : 82%|████████▏ | 2.98GB / 3.63GB, 266MB/s ...ve-03b89ee9/checkpoint_15: 92%|█████████▏| 3.35GB / 3.63GB  Processing Files (0 / 1) : 92%|█████████▏| 3.35GB / 3.63GB, 294MB/s ...ve-03b89ee9/checkpoint_15: 100%|██████████| 3.63GB / 3.63GB  Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s New Data Upload : | | 0.00B / 0.00B, 0.00B/s ...ve-03b89ee9/checkpoint_15: 100%|██████████| 3.63GB / 3.63GB No files have been modified since last commit. Skipping to prevent empty commit. Uploaded checkpoint_15 to seyedparsa/thoughtformer-checkpoints/checkpoints/star-datacurr-only-4gpu-bs32-lr1e-5_n1000_qwen3-0.6b-base_thoughtformer_lr1e-5_no-reset_dc_no-tc_adaptive-03b89ee9/checkpoint_15 ============================================================ EPOCH 15/100 (thought_stage=10, data_stage=9) ============================================================ thought_stage=10, c_thought=1, max_difficulty=9 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 10 *** Processing Files (0 / 0) : | | 0.00B / 0.00B New Data Upload : | | 0.00B / 0.00B  ...ve-03b89ee9/checkpoint_16: 10%|▉ | 352MB / 3.63GB  ...ve-03b89ee9/checkpoint_16: 10%|▉ | 352MB / 3.63GB  Processing Files (0 / 1) : 10%|▉ | 352MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_16: 20%|█▉ | 720MB / 3.63GB  Processing Files (0 / 1) : 20%|█▉ | 720MB / 3.63GB, ???B/s ...ve-03b89ee9/checkpoint_16: 30%|███ | 1.10GB / 3.63GB  Processing Files (0 / 1) : 30%|███ | 1.10GB / 3.63GB, 105MB/s ...ve-03b89ee9/checkpoint_16: 41%|████ | 1.48GB / 3.63GB  Processing Files (0 / 1) : 41%|████ | 1.48GB / 3.63GB, 140MB/s ...ve-03b89ee9/checkpoint_16: 51%|█████ | 1.86GB / 3.63GB  Processing Files (0 / 1) : 51%|█████ | 1.86GB / 3.63GB, 173MB/s ...ve-03b89ee9/checkpoint_16: 62%|██████▏ | 2.23GB / 3.63GB  Processing Files (0 / 1) : 62%|██████▏ | 2.23GB / 3.63GB, 205MB/s ...ve-03b89ee9/checkpoint_16: 72%|███████▏ | 2.61GB / 3.63GB  Processing Files (0 / 1) : 72%|███████▏ | 2.61GB / 3.63GB, 236MB/s ...ve-03b89ee9/checkpoint_16: 82%|████████▏ | 2.98GB / 3.63GB  Processing Files (0 / 1) : 82%|████████▏ | 2.98GB / 3.63GB, 266MB/s ...ve-03b89ee9/checkpoint_16: 93%|█████████▎| 3.36GB / 3.63GB  Processing Files (0 / 1) : 93%|█████████▎| 3.36GB / 3.63GB, 295MB/s ...ve-03b89ee9/checkpoint_16: 100%|██████████| 3.63GB / 3.63GB  Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s Processing Files (1 / 1) : 100%|██████████| 3.63GB / 3.63GB, 313MB/s New Data Upload : | | 0.00B / 0.00B, 0.00B/s ...ve-03b89ee9/checkpoint_16: 100%|██████████| 3.63GB / 3.63GB No files have been modified since last commit. Skipping to prevent empty commit. Uploaded checkpoint_16 to seyedparsa/thoughtformer-checkpoints/checkpoints/star-datacurr-only-4gpu-bs32-lr1e-5_n1000_qwen3-0.6b-base_thoughtformer_lr1e-5_no-reset_dc_no-tc_adaptive-03b89ee9/checkpoint_16 ============================================================ EPOCH 16/100 (thought_stage=10, data_stage=10) ============================================================ thought_stage=10, c_thought=1, max_difficulty=10 Map (num_proc=4): 0%| | 0/1000 [00:00