W0530 22:18:22.514000 714703 torch/distributed/run.py:862] W0530 22:18:22.514000 714703 torch/distributed/run.py:862] ***************************************** W0530 22:18:22.514000 714703 torch/distributed/run.py:862] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W0530 22:18:22.514000 714703 torch/distributed/run.py:862] ***************************************** Config: {'project': 'thoughtformer', 'name': 'star-atc-4gpu-bs32-lr1e-5-fixed', 'only_eval': False, 'method': 'thoughtformer', 'data_curriculum': True, 'thought_curriculum': True, 'staging': 'adaptive', 'init_thought_stage': 1, 'init_data_stage': 1, 'patience': 5, 'staging_threshold': 0.9, 'c_thought': 1, 'max_latent_stage': 10, 'uniform_prob': 0.1, 'save_only_improve': False, 'model_id': 'Qwen/Qwen3-0.6B-Base', 'load_model_path': 'None', 'seed': 0, 'resume': 0, 'bf16': False, 'train_path': 'data/star_k10_L10_1000_train.json', 'val_path': 'data/star_k10_L10_100_valid.json', 'reset_optimizer': False, 'lr_schedule': 'none', 'batch_size_training': 4, 'eval_only_trained': False, 'eval_batch_size': 8, 'eval_every': 1, 'debug': False, 'gradient_accumulation_steps': 2, 'num_epochs': 100, 'lr': 1e-05, 'weight_decay': 0.01, 'group': 'star_atc_fixed_lr', 'tags': ['star', 'k10', 'L10', 'atc', 'adaptive', 'lr1e-5', 'best', 'budget_aug'], 'run_type': 'pilot'} /home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. return func(*args, **kwargs) [rank0]:[W530 22:18:27.516767397 ProcessGroupNCCL.cpp:5324] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group() Loading weights: 0%| | 0/310 [00:00= 0.9 *** *** Advancing to stage 2 *** Warning: failed to upload checkpoint_1 to HF: (Request ID: Root=1-6a1b6395-04a825f05758e3e428a2aa03;20d24efa-aa1d-48ea-ace8-e1c4fd155f80) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 1/100 (thought_stage=2, data_stage=2) ============================================================ thought_stage=2, c_thought=1, max_difficulty=2 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 3 *** Warning: failed to upload checkpoint_2 to HF: (Request ID: Root=1-6a1b6525-03e546bd213c0e8d2d2d47a2;1c35a3e8-eec8-4dd1-8f3a-28bea12c616d) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 2/100 (thought_stage=3, data_stage=3) ============================================================ thought_stage=3, c_thought=1, max_difficulty=3 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 4 *** Warning: failed to upload checkpoint_3 to HF: (Request ID: Root=1-6a1b6722-31eeacfc16182155080bf1ee;335cdbbe-fd02-4632-b033-2cb065d3800e) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 3/100 (thought_stage=4, data_stage=4) ============================================================ thought_stage=4, c_thought=1, max_difficulty=4 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 5 *** Warning: failed to upload checkpoint_4 to HF: (Request ID: Root=1-6a1b69b1-2ab9d377552f7ed766a59c7b;e55c7a49-1f86-4d9b-aa0b-6693908daa8b) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 4/100 (thought_stage=5, data_stage=5) ============================================================ thought_stage=5, c_thought=1, max_difficulty=5 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 6 *** Warning: failed to upload checkpoint_5 to HF: (Request ID: Root=1-6a1b6cf4-7078d5db257f545f3ae07a8e;c8a4a3b7-4ca4-4522-ab79-f066430700a4) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 5/100 (thought_stage=6, data_stage=6) ============================================================ thought_stage=6, c_thought=1, max_difficulty=6 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 7 *** Warning: failed to upload checkpoint_6 to HF: (Request ID: Root=1-6a1b710d-55cc61d01da02b0234ebc71e;a30dcadf-f3a1-4f0d-8601-0a8ead664deb) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 6/100 (thought_stage=7, data_stage=7) ============================================================ thought_stage=7, c_thought=1, max_difficulty=7 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 8 *** Warning: failed to upload checkpoint_7 to HF: (Request ID: Root=1-6a1b7620-29176ede23ed252165556f86;75876f1e-62ac-4312-b53b-7c47b1c12d57) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 7/100 (thought_stage=8, data_stage=8) ============================================================ thought_stage=8, c_thought=1, max_difficulty=8 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 9 *** Warning: failed to upload checkpoint_8 to HF: (Request ID: Root=1-6a1b7c4a-027e0cdc1e917c7f118e3dbb;02f263b4-1644-4aea-b7f6-2e3d3b6a8366) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 8/100 (thought_stage=9, data_stage=9) ============================================================ thought_stage=9, c_thought=1, max_difficulty=9 Map (num_proc=4): 0%| | 0/1000 [00:00= 0.9 *** *** Advancing to stage 10 *** Warning: failed to upload checkpoint_9 to HF: (Request ID: Root=1-6a1b83c4-17125c1b184cc12d05791a2d;cb320e8e-b4d4-4302-860e-3724fd216c0c) 403 Forbidden: You need to setup automatic credit recharge in order to upload more data. You can do so at /settings/billing.. Cannot access content at: https://huggingface.co/seyedparsa/thoughtformer-checkpoints.git/info/lfs/objects/batch. Make sure your token has the correct permissions. ============================================================ EPOCH 9/100 (thought_stage=10, data_stage=10) ============================================================ thought_stage=10, c_thought=1, max_difficulty=10 Map (num_proc=4): 0%| | 0/1000 [00:00 sys.exit(main()) ^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 367, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/run.py", line 1016, in main run(args) File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/run.py", line 1007, in run elastic_launch( File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 191, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 362, in launch_agent result = agent.run() ^^^^^^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper result = f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 745, in run result = self._invoke_run(role) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/elastic/agent/server/api.py", line 923, in _invoke_run time.sleep(monitor_interval) File "/home/ubuntu/thoughtformer/.venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval) torch.distributed.elastic.multiprocessing.api.SignalException: Process 714703 got signal: 15