[2025-11-13 08:04:10,154][mllm.models.large_language_model_local][INFO] - Initializing adapter 'agent_adapter': no initial weights provided or found; starting from scratch.
[2025-11-13 08:04:10,973][mllm.models.adapter_training_wrapper][INFO] - Adapter 'agent_adapter': initialized with fresh weights (no initial weights found).
[2025-11-13 08:04:10,980][mllm.models.large_language_model_local][INFO] - Initializing adapter 'critic_adapter': no initial weights provided or found; starting from scratch.
[2025-11-13 08:04:12,041][mllm.models.adapter_training_wrapper][INFO] - Adapter 'critic_adapter': initialized with fresh weights (no initial weights found).
[2025-11-13 08:06:23,278][__main__][INFO] - Starting iteration 0.
[2025-11-13 08:06:23,282][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:06:23,283][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:06:28,136][__main__][INFO] - Number of regex retries in iteration 0: 0
[2025-11-13 08:06:28,137][__main__][INFO] - agents played in iteration 0 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:06:28,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,691][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:06:28,692][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:06:29,360][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:06:29,985][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:06:30,312][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:06:30,636][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:06:30,970][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:06:31,301][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:06:31,635][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:06:31,966][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:06:32,296][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:06:32,624][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:06:32,955][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:06:33,287][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:06:33,615][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:06:33,948][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:06:34,281][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:06:34,615][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:06:34,938][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:06:35,266][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:06:35,594][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:06:35,925][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:06:36,250][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:06:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:06:36,907][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:06:37,236][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:06:37,563][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:06:37,888][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:06:38,212][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:06:38,538][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:06:38,866][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:06:39,190][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:06:39,515][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:06:39,839][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:06:40,164][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:06:40,862][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 4.58%, Current % of VRAM taken: 42.03%, Block Peak % of device VRAM: 25.21%, ΔTime: 00:00:11
[2025-11-13 08:06:41,549][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:06:41,550][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:06:41,552][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:06:42,792][__main__][INFO] - Iteration 1 took 19s (24.88% Gen, 68.76% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 12m 37s. Estimated total time: 16h 15m 31s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 31s, 500 more iterations: 2h 42m 35s.
[2025-11-13 08:06:42,794][__main__][INFO] - Starting iteration 1.
[2025-11-13 08:06:42,797][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:06:42,798][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:06:44,822][mllm.models.large_language_model_local][WARNING] - Response %A> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 08:06:47,159][__main__][INFO] - Number of regex retries in iteration 1: 1
[2025-11-13 08:06:47,160][__main__][INFO] - agents played in iteration 1 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:06:47,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,726][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:06:47,727][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:06:48,375][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:06:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:06:48,994][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:06:49,316][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:06:49,638][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:06:49,961][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:06:50,283][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:06:50,606][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:06:50,930][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:06:51,259][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:06:51,584][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:06:51,906][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:06:52,228][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:06:52,549][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:06:52,871][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:06:53,195][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:06:53,517][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:06:53,841][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:06:54,162][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:06:54,485][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:06:54,808][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:06:55,131][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:06:55,455][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:06:55,778][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:06:56,105][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:06:56,428][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:06:56,750][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:06:57,072][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:06:57,401][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:06:57,722][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:06:58,045][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:06:58,367][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:06:58,693][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:06:59,393][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:00,130][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:00,132][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:00,134][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:01,231][__main__][INFO] - Iteration 2 took 18s (23.66% Gen, 70.38% Train). Generation: 4s, Training: 12s. Estimated remaining time: 15h 18m 33s. Estimated total time: 15h 21m 45s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 43s, 500 more iterations: 2h 33m 37s.
[2025-11-13 08:07:01,233][__main__][INFO] - Starting iteration 2.
[2025-11-13 08:07:01,236][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:01,236][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:05,055][__main__][INFO] - Number of regex retries in iteration 2: 0
[2025-11-13 08:07:05,056][__main__][INFO] - agents played in iteration 2 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:07:05,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,616][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:05,616][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:06,281][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:06,587][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:06,910][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:07,559][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:07,893][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:08,217][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:08,539][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:08,861][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:09,197][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:09,522][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:09,846][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:10,170][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:10,501][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:10,825][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:11,152][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:11,475][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:11,803][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:12,126][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:12,453][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:12,782][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:13,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:13,441][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:13,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:14,091][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:14,420][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:14,745][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:15,070][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:15,397][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:15,717][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:16,042][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:16,372][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:16,703][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:17,419][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:18,175][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:18,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:18,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:19,131][__main__][INFO] - Iteration 3 took 17s (21.34% Gen, 73.32% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 51m 17s. Estimated total time: 14h 54m 47s. Time estimates for 10 more iterations: 2m 58s, 100 more iterations: 29m 49s, 500 more iterations: 2h 29m 7s.
[2025-11-13 08:07:19,133][__main__][INFO] - Starting iteration 3.
[2025-11-13 08:07:19,135][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:19,136][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:22,959][__main__][INFO] - Number of regex retries in iteration 3: 0
[2025-11-13 08:07:22,960][__main__][INFO] - agents played in iteration 3 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:07:23,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,465][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,499][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,533][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:23,533][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:24,221][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:24,516][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:24,839][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:25,162][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:25,486][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:25,810][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:26,136][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:26,459][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:26,782][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:27,105][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:27,428][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:28,076][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:28,401][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:28,723][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:29,046][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:29,369][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:29,692][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:30,016][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:30,340][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:30,663][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:30,986][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:31,312][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:31,636][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:31,960][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:32,287][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:32,610][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:32,934][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:33,257][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:33,581][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:33,904][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:34,228][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:34,552][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:35,260][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:35,996][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:35,997][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:35,999][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:37,000][__main__][INFO] - Iteration 4 took 17s (21.40% Gen, 72.99% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 49m 29s. Estimated total time: 14h 53m 17s. Time estimates for 10 more iterations: 2m 58s, 100 more iterations: 29m 46s, 500 more iterations: 2h 28m 52s.
[2025-11-13 08:07:37,002][__main__][INFO] - Starting iteration 4.
[2025-11-13 08:07:37,005][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:37,006][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:40,791][__main__][INFO] - Number of regex retries in iteration 4: 0
[2025-11-13 08:07:40,792][__main__][INFO] - agents played in iteration 4 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:07:41,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,341][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:41,341][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:42,057][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:42,352][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:42,675][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:42,998][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:43,322][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:43,646][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:43,971][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:44,299][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:44,942][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:45,265][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:45,597][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:45,917][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:46,240][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:46,564][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:47,212][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:47,536][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:47,858][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:48,185][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:48,510][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:48,833][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:49,157][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:49,815][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:50,139][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:50,467][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:50,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:51,114][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:51,438][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:51,762][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:52,088][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:52,410][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:53,142][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:53,916][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:53,918][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:53,920][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:54,955][__main__][INFO] - Iteration 5 took 17s (21.09% Gen, 73.14% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 53m 24s. Estimated total time: 14h 57m 30s. Time estimates for 10 more iterations: 2m 59s, 100 more iterations: 29m 55s, 500 more iterations: 2h 29m 35s.
[2025-11-13 08:07:54,957][__main__][INFO] - Starting iteration 5.
[2025-11-13 08:07:54,960][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:54,960][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:58,749][__main__][INFO] - Number of regex retries in iteration 5: 0
[2025-11-13 08:07:58,750][__main__][INFO] - agents played in iteration 5 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:07:59,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,278][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,310][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:59,311][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:00,026][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:00,321][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:00,645][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:00,967][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:01,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:01,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:01,935][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:02,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:02,580][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:02,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:03,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:03,552][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:03,876][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:04,199][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:04,520][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:04,843][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:05,165][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:05,815][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:06,140][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:06,464][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:06,789][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:07,111][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:07,434][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:07,758][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:08,081][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:08,410][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:08,735][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:09,058][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:09,383][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:09,708][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:10,031][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:10,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:11,091][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:11,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:11,840][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:11,842][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:12,871][__main__][INFO] - Iteration 6 took 17s (21.16% Gen, 73.09% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 51m 11s. Estimated total time: 14h 55m 35s. Time estimates for 10 more iterations: 2m 59s, 100 more iterations: 29m 51s, 500 more iterations: 2h 29m 15s.
[2025-11-13 08:08:12,873][__main__][INFO] - Starting iteration 6.
[2025-11-13 08:08:12,877][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:12,877][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:16,769][__main__][INFO] - Number of regex retries in iteration 6: 0
[2025-11-13 08:08:16,769][__main__][INFO] - agents played in iteration 6 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:08:17,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,309][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,342][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:17,343][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:18,059][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:18,354][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:18,678][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:19,001][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:19,324][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:19,647][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:19,969][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:20,292][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:20,614][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:20,938][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:21,266][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:21,589][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:21,912][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:22,238][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:22,562][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:22,884][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:23,207][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:23,531][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:23,855][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:24,178][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:24,501][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:24,824][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:25,146][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:25,468][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:25,792][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:26,115][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:26,439][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:26,762][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:27,086][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:27,409][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:27,736][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:28,064][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:28,388][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:29,131][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:29,862][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:29,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:29,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:31,057][__main__][INFO] - Iteration 7 took 18s (21.41% Gen, 72.03% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 4m 23s. Estimated total time: 15h 9m 5s. Time estimates for 10 more iterations: 3m 1s, 100 more iterations: 30m 18s, 500 more iterations: 2h 31m 30s.
[2025-11-13 08:08:31,059][__main__][INFO] - Starting iteration 7.
[2025-11-13 08:08:31,063][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:31,063][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:34,895][__main__][INFO] - Number of regex retries in iteration 7: 0
[2025-11-13 08:08:34,896][__main__][INFO] - agents played in iteration 7 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:08:35,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,447][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:35,448][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:36,185][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:36,480][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:36,804][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:37,132][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:37,450][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:37,772][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:38,096][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:38,420][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:38,744][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:39,069][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:39,392][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:39,717][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:40,040][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:40,364][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:40,686][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:41,018][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:41,332][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:41,655][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:41,980][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:42,303][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:42,625][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:42,948][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:43,272][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:43,596][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:43,920][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:44,243][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:44,566][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:44,890][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:45,218][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:45,544][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:45,869][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:46,193][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:46,516][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:47,252][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:48,007][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:48,008][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:48,010][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:49,097][__main__][INFO] - Iteration 8 took 18s (21.25% Gen, 72.72% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 56m 44s. Estimated total time: 15h 1m 44s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 3s, 500 more iterations: 2h 30m 17s.
[2025-11-13 08:08:49,099][__main__][INFO] - Starting iteration 8.
[2025-11-13 08:08:49,102][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:49,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:53,003][__main__][INFO] - Number of regex retries in iteration 8: 0
[2025-11-13 08:08:53,003][__main__][INFO] - agents played in iteration 8 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:08:53,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,498][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,563][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:53,564][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:54,313][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:54,607][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:54,931][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:55,253][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:55,577][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:55,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:56,224][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:56,548][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:56,871][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:57,206][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:57,530][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:57,853][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:58,177][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:58,500][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:58,825][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:59,150][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:59,476][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:59,801][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:00,131][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:00,461][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:00,784][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:01,430][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:01,752][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:02,075][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:02,411][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:02,734][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:03,063][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:03,386][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:03,717][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:04,039][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:04,363][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:04,686][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:05,432][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:06,217][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:06,218][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:06,220][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:07,246][__main__][INFO] - Iteration 9 took 18s (21.49% Gen, 72.84% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 1m 54s. Estimated total time: 15h 7m 12s. Time estimates for 10 more iterations: 3m 1s, 100 more iterations: 30m 14s, 500 more iterations: 2h 31m 12s.
[2025-11-13 08:09:07,248][__main__][INFO] - Starting iteration 9.
[2025-11-13 08:09:07,251][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:09:07,251][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:11,057][__main__][INFO] - Number of regex retries in iteration 9: 0
[2025-11-13 08:09:11,058][__main__][INFO] - agents played in iteration 9 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:09:11,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,627][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:11,627][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:12,357][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:12,656][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:12,977][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:13,300][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:13,626][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:13,954][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:14,283][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:14,610][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:14,935][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:15,259][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:15,584][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:15,910][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:16,235][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:16,563][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:16,894][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:17,220][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:17,546][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:17,875][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:18,210][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:18,534][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:18,857][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:19,183][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:19,521][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:19,850][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:20,176][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:20,501][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:20,829][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:21,153][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:21,478][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:21,804][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:22,128][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:22,451][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:22,779][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:23,542][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:24,286][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:24,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:24,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:25,296][__main__][INFO] - Iteration 10 took 18s (21.10% Gen, 73.32% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 56m 39s. Estimated total time: 15h 2m 16s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 4s, 500 more iterations: 2h 30m 22s.
[2025-11-13 08:09:25,298][__main__][INFO] - Starting iteration 10.
[2025-11-13 08:09:25,301][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:09:25,301][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:29,169][__main__][INFO] - Number of regex retries in iteration 10: 0
[2025-11-13 08:09:29,170][__main__][INFO] - agents played in iteration 10 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:09:29,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,731][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:29,731][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:30,482][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:30,779][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:31,102][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:31,432][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:31,761][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:32,091][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:32,416][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:32,742][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:33,068][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:33,391][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:33,716][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:34,043][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:34,369][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:34,691][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:35,015][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:35,345][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:35,663][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:35,988][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:36,311][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:36,642][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:36,959][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:37,285][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:37,611][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:37,944][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:38,264][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:38,594][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:38,919][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:39,254][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:39,571][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:39,893][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:40,217][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:40,546][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:40,866][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:41,571][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:42,313][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:42,314][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:42,316][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:44,404][__main__][INFO] - Iteration 11 took 19s (20.25% Gen, 68.82% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 49m 16s. Estimated total time: 15h 55m 12s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 50s, 500 more iterations: 2h 39m 12s.
[2025-11-13 08:09:44,406][__main__][INFO] - Starting iteration 11.
[2025-11-13 08:09:44,410][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:09:44,410][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:48,907][__main__][INFO] - Number of regex retries in iteration 11: 0
[2025-11-13 08:09:48,907][__main__][INFO] - agents played in iteration 11 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:09:49,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,473][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:49,473][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:50,231][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:50,526][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:50,852][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:51,179][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:51,504][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:51,829][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:52,158][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:52,482][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:52,810][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:53,136][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:53,461][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:53,784][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:54,108][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:54,430][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:54,753][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:55,076][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:55,398][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:55,722][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:56,047][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:56,370][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:56,693][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:57,017][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:57,343][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:57,666][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:57,990][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:58,312][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:58,636][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:58,962][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:59,286][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:59,613][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:59,940][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:00,265][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:00,589][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:01,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:02,079][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:02,082][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:02,083][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:03,066][__main__][INFO] - Iteration 12 took 18s (24.10% Gen, 70.62% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 26m 37s. Estimated total time: 15h 32m 51s. Time estimates for 10 more iterations: 3m 6s, 100 more iterations: 31m 5s, 500 more iterations: 2h 35m 28s.
[2025-11-13 08:10:03,068][__main__][INFO] - Starting iteration 12.
[2025-11-13 08:10:03,071][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:03,072][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:07,422][__main__][INFO] - Number of regex retries in iteration 12: 0
[2025-11-13 08:10:07,423][__main__][INFO] - agents played in iteration 12 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:10:07,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:07,918][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:07,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:07,984][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:07,985][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:07,985][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:08,744][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:09,041][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:09,368][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:09,695][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:10,025][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:10,348][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:10,671][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:10,996][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:11,319][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:11,644][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:11,969][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:12,296][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:12,621][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:12,945][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:13,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:13,595][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:13,919][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:14,247][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:14,572][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:14,898][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:15,227][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:15,557][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:15,881][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:16,204][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:16,528][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:16,852][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:17,176][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:17,499][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:17,823][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:18,148][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:18,471][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:18,795][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:19,119][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:19,873][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:20,642][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:20,644][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:20,646][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:21,683][__main__][INFO] - Iteration 13 took 18s (23.38% Gen, 71.05% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 24m 4s. Estimated total time: 15h 30m 37s. Time estimates for 10 more iterations: 3m 6s, 100 more iterations: 31m 1s, 500 more iterations: 2h 35m 6s.
[2025-11-13 08:10:21,685][__main__][INFO] - Starting iteration 13.
[2025-11-13 08:10:21,688][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:21,689][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:25,873][__main__][INFO] - Number of regex retries in iteration 13: 0
[2025-11-13 08:10:25,873][__main__][INFO] - agents played in iteration 13 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:10:26,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,441][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:26,441][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:27,192][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:27,486][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:27,810][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:28,134][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:28,457][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:28,780][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:29,104][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:29,435][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:29,759][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:30,081][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:30,404][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:30,735][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:31,061][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:31,385][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:31,708][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:32,038][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:32,361][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:32,685][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:33,008][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:33,334][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:33,658][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:33,983][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:34,308][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:34,637][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:34,964][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:35,287][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:35,612][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:35,941][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:36,271][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:36,597][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:36,923][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:37,248][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:37,570][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:38,348][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:39,099][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:39,100][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:39,102][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:40,095][__main__][INFO] - Iteration 14 took 18s (22.73% Gen, 71.87% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 13m 29s. Estimated total time: 15h 20m 21s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 40s, 500 more iterations: 2h 33m 23s.
[2025-11-13 08:10:40,098][__main__][INFO] - Starting iteration 14.
[2025-11-13 08:10:40,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:40,102][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:44,305][__main__][INFO] - Number of regex retries in iteration 14: 0
[2025-11-13 08:10:44,306][__main__][INFO] - agents played in iteration 14 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:10:44,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,872][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:44,872][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:45,619][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:45,914][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:46,238][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:46,561][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:47,213][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:47,539][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:47,863][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:48,187][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:48,514][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:48,837][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:49,167][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:49,490][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:49,815][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:50,139][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:50,462][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:50,789][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:51,113][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:51,437][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:51,761][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:52,086][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:52,412][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:52,745][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:53,065][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:53,389][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:53,714][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:54,046][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:54,362][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:54,686][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:55,009][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:55,338][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:55,983][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:56,730][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:57,483][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:57,484][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:57,486][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:58,475][__main__][INFO] - Iteration 15 took 18s (22.88% Gen, 71.73% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 11m 34s. Estimated total time: 15h 18m 43s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 37s, 500 more iterations: 2h 33m 7s.
[2025-11-13 08:10:58,477][__main__][INFO] - Starting iteration 15.
[2025-11-13 08:10:58,481][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:58,482][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:02,775][__main__][INFO] - Number of regex retries in iteration 15: 0
[2025-11-13 08:11:02,776][__main__][INFO] - agents played in iteration 15 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:11:03,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,331][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,364][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:03,365][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:04,114][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:04,410][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:04,738][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:05,060][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:05,388][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:05,715][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:06,046][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:06,368][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:06,692][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:07,016][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:07,340][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:07,666][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:07,991][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:08,314][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:08,639][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:08,964][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:09,287][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:09,610][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:09,941][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:10,264][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:10,587][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:11,244][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:11,567][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:11,889][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:12,214][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:12,545][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:12,867][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:13,190][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:13,513][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:13,845][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:14,168][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:14,491][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:15,249][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:15,996][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:15,998][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:16,000][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:17,026][__main__][INFO] - Iteration 16 took 18s (23.15% Gen, 71.31% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 19m 48s. Estimated total time: 15h 27m 16s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 54s, 500 more iterations: 2h 34m 32s.
[2025-11-13 08:11:17,028][__main__][INFO] - Starting iteration 16.
[2025-11-13 08:11:17,032][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:17,032][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:21,251][__main__][INFO] - Number of regex retries in iteration 16: 0
[2025-11-13 08:11:21,251][__main__][INFO] - agents played in iteration 16 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:11:21,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,758][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,791][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,824][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,825][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:21,825][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:22,588][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:22,883][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:23,207][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:23,531][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:23,855][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:24,179][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:24,825][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:25,148][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:25,472][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:25,797][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:26,120][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:26,444][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:26,768][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:27,091][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:27,415][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:27,738][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:28,062][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:28,389][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:28,712][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:29,037][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:29,362][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:29,685][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:30,012][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:30,337][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:30,664][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:30,990][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:31,316][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:31,639][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:31,962][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:32,285][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:32,608][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:32,935][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:33,682][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:34,432][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:34,433][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:34,435][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:35,434][__main__][INFO] - Iteration 17 took 18s (22.93% Gen, 71.64% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 12m 22s. Estimated total time: 15h 20m 9s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 40s, 500 more iterations: 2h 33m 21s.
[2025-11-13 08:11:35,436][__main__][INFO] - Starting iteration 17.
[2025-11-13 08:11:35,439][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:35,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:39,769][__main__][INFO] - Number of regex retries in iteration 17: 0
[2025-11-13 08:11:39,769][__main__][INFO] - agents played in iteration 17 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:11:40,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,356][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:40,356][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:41,088][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:41,384][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:41,710][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:42,032][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:42,355][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:42,679][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:43,003][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:43,326][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:43,649][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:43,971][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:44,295][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:44,942][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:45,266][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:45,589][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:45,911][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:46,235][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:46,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:46,883][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:47,205][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:47,529][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:47,852][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:48,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:48,499][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:48,823][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:49,147][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:49,470][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:49,793][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:50,116][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:50,439][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:50,762][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:51,085][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:51,409][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:52,156][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:52,895][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:52,897][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:52,898][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:53,899][__main__][INFO] - Iteration 18 took 18s (23.45% Gen, 71.12% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 14m 56s. Estimated total time: 15h 23m 1s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 46s, 500 more iterations: 2h 33m 50s.
[2025-11-13 08:11:53,901][__main__][INFO] - Starting iteration 18.
[2025-11-13 08:11:53,904][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:53,904][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:58,163][__main__][INFO] - Number of regex retries in iteration 18: 0
[2025-11-13 08:11:58,163][__main__][INFO] - agents played in iteration 18 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:11:58,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,737][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:58,738][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:59,488][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:59,784][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:00,108][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:00,434][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:00,766][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:01,082][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:01,405][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:01,730][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:02,053][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:02,377][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:02,700][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:03,025][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:03,349][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:03,671][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:03,995][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:04,318][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:04,643][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:04,966][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:05,289][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:05,613][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:05,937][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:06,262][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:06,587][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:06,911][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:07,234][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:07,558][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:07,881][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:08,207][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:08,531][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:08,854][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:09,179][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:09,501][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:09,823][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:10,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:11,347][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:11,348][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:11,350][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:12,428][__main__][INFO] - Iteration 19 took 18s (22.99% Gen, 71.18% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 17m 52s. Estimated total time: 15h 26m 15s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 52s, 500 more iterations: 2h 34m 22s.
[2025-11-13 08:12:12,430][__main__][INFO] - Starting iteration 19.
[2025-11-13 08:12:12,434][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:12:12,434][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:16,674][__main__][INFO] - Number of regex retries in iteration 19: 0
[2025-11-13 08:12:16,674][__main__][INFO] - agents played in iteration 19 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:12:17,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,243][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:17,243][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:17,968][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:18,263][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:18,588][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:18,911][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:19,235][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:19,883][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:20,206][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:20,528][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:20,864][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:21,188][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:21,512][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:21,835][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:22,158][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:22,481][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:22,805][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:23,131][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:23,464][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:23,787][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:24,110][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:24,433][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:24,757][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:25,081][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:25,404][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:26,053][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:26,379][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:26,702][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:27,025][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:27,359][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:27,681][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:28,005][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:28,329][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:29,091][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:29,850][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:29,852][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:29,853][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:30,889][__main__][INFO] - Iteration 20 took 18s (22.97% Gen, 71.41% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 14m 6s. Estimated total time: 15h 22m 49s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 45s, 500 more iterations: 2h 33m 48s.
[2025-11-13 08:12:30,891][__main__][INFO] - Starting iteration 20.
[2025-11-13 08:12:30,894][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:12:30,895][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:35,115][__main__][INFO] - Number of regex retries in iteration 20: 0
[2025-11-13 08:12:35,116][__main__][INFO] - agents played in iteration 20 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:12:35,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,694][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:35,695][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:36,447][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:36,744][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:37,068][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:37,394][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:37,720][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:38,045][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:38,367][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:38,690][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:39,014][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:39,338][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:39,661][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:39,985][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:40,309][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:40,633][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:40,955][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:41,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:41,604][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:41,926][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:42,249][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:42,572][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:42,896][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:43,221][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:43,548][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:43,870][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:44,199][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:44,517][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:45,164][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:45,487][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:45,811][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:46,135][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:46,459][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:46,783][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:47,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:48,287][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:48,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:48,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:50,333][__main__][INFO] - Iteration 21 took 19s (21.72% Gen, 67.77% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 2m 56s. Estimated total time: 16h 11m 57s. Time estimates for 10 more iterations: 3m 14s, 100 more iterations: 32m 23s, 500 more iterations: 2h 41m 59s.
[2025-11-13 08:12:50,335][__main__][INFO] - Starting iteration 21.
[2025-11-13 08:12:50,337][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:12:50,338][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:54,649][__main__][INFO] - Number of regex retries in iteration 21: 0
[2025-11-13 08:12:54,650][__main__][INFO] - agents played in iteration 21 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:12:55,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,221][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:55,221][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:55,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:56,256][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:56,581][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:56,906][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:57,231][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:57,557][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:57,884][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:58,210][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:58,535][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:58,860][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:59,184][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:59,507][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:59,833][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:00,159][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:00,484][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:00,808][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:01,130][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:01,456][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:01,779][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:02,102][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:02,425][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:02,748][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:03,072][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:03,395][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:03,718][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:04,040][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:04,365][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:04,691][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:05,016][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:05,343][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:05,666][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:05,989][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:06,314][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:07,033][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:07,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:07,769][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:07,773][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:08,764][__main__][INFO] - Iteration 22 took 18s (23.40% Gen, 71.22% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 12m 1s. Estimated total time: 15h 21m 21s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 42s, 500 more iterations: 2h 33m 33s.
[2025-11-13 08:13:08,766][__main__][INFO] - Starting iteration 22.
[2025-11-13 08:13:08,769][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:08,770][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:13,155][__main__][INFO] - Number of regex retries in iteration 22: 0
[2025-11-13 08:13:13,156][__main__][INFO] - agents played in iteration 22 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:13:13,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,740][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:13,741][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:14,485][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:14,787][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:15,110][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:15,433][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:15,755][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:16,078][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:16,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:16,725][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:17,049][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:17,381][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:17,704][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:18,028][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:18,351][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:18,675][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:18,999][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:19,322][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:19,646][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:19,980][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:20,303][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:20,627][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:20,951][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:21,278][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:21,602][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:21,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:22,249][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:22,581][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:22,906][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:23,230][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:23,553][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:23,885][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:24,209][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:24,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:24,855][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:25,596][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:26,362][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:26,363][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:26,365][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:27,377][__main__][INFO] - Iteration 23 took 18s (23.57% Gen, 70.98% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 20m 48s. Estimated total time: 15h 30m 27s. Time estimates for 10 more iterations: 3m 6s, 100 more iterations: 31m 0s, 500 more iterations: 2h 35m 4s.
[2025-11-13 08:13:27,379][__main__][INFO] - Starting iteration 23.
[2025-11-13 08:13:27,383][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:27,383][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:31,614][__main__][INFO] - Number of regex retries in iteration 23: 0
[2025-11-13 08:13:31,614][__main__][INFO] - agents played in iteration 23 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:13:32,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,206][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:32,206][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:32,983][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:33,278][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:33,603][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:33,927][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:34,251][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:34,573][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:34,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:35,222][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:35,548][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:35,872][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:36,196][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:36,520][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:36,843][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:37,167][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:37,489][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:37,813][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:38,137][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:38,463][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:38,793][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:39,119][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:39,443][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:39,772][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:40,096][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:40,419][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:40,744][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:41,067][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:41,391][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:41,714][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:42,041][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:42,366][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:42,693][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:43,016][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:43,341][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:44,083][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:44,843][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:44,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:44,846][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:45,861][__main__][INFO] - Iteration 24 took 18s (22.90% Gen, 71.60% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 14m 0s. Estimated total time: 15h 23m 57s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 47s, 500 more iterations: 2h 33m 59s.
[2025-11-13 08:13:45,864][__main__][INFO] - Starting iteration 24.
[2025-11-13 08:13:45,867][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:45,868][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:50,052][__main__][INFO] - Number of regex retries in iteration 24: 0
[2025-11-13 08:13:50,053][__main__][INFO] - agents played in iteration 24 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:13:50,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,644][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:50,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:51,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:51,696][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:52,021][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:52,345][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:52,681][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:53,005][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:53,328][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:53,655][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:53,991][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:54,315][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:54,640][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:54,964][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:55,297][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:55,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:55,953][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:56,280][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:56,605][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:56,929][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:57,254][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:57,580][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:57,906][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:58,237][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:58,563][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:58,895][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:59,213][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:59,537][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:59,865][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:00,192][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:00,513][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:00,840][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:01,166][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:01,492][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:01,821][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:02,568][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:03,308][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:03,309][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:03,311][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:04,328][__main__][INFO] - Iteration 25 took 18s (22.67% Gen, 71.81% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 12m 49s. Estimated total time: 15h 23m 4s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 46s, 500 more iterations: 2h 33m 50s.
[2025-11-13 08:14:04,330][__main__][INFO] - Starting iteration 25.
[2025-11-13 08:14:04,333][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:04,334][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:08,558][__main__][INFO] - Number of regex retries in iteration 25: 0
[2025-11-13 08:14:08,559][__main__][INFO] - agents played in iteration 25 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:14:09,020][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,121][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:09,122][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:09,882][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:10,177][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:10,502][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:10,830][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:11,153][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:11,476][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:11,801][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:12,127][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:12,449][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:12,773][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:13,096][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:13,424][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:13,747][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:14,070][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:14,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:15,044][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:15,368][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:15,691][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:16,341][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:16,667][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:16,991][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:17,314][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:17,638][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:17,961][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:18,285][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:18,608][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:18,932][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:19,257][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:19,581][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:19,904][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:20,229][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:20,960][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:21,743][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:21,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:21,746][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:22,759][__main__][INFO] - Iteration 26 took 18s (22.93% Gen, 71.56% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 10m 47s. Estimated total time: 15h 21m 21s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 42s, 500 more iterations: 2h 33m 33s.
[2025-11-13 08:14:22,762][__main__][INFO] - Starting iteration 26.
[2025-11-13 08:14:22,766][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:22,766][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:27,074][__main__][INFO] - Number of regex retries in iteration 26: 0
[2025-11-13 08:14:27,074][__main__][INFO] - agents played in iteration 26 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:14:27,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:27,649][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:28,397][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:28,693][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:29,022][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:29,343][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:29,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:29,989][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:30,315][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:30,637][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:30,958][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:31,281][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:31,603][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:31,927][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:32,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:32,576][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:32,905][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:33,230][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:33,553][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:33,876][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:34,199][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:34,849][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:35,171][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:35,494][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:35,817][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:36,142][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:36,464][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:36,789][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:37,112][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:37,435][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:37,757][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:38,082][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:38,405][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:38,730][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:39,463][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:40,206][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:40,207][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:40,209][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:41,203][__main__][INFO] - Iteration 27 took 18s (23.37% Gen, 71.24% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 11m 2s. Estimated total time: 15h 21m 54s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 43s, 500 more iterations: 2h 33m 39s.
[2025-11-13 08:14:41,206][__main__][INFO] - Starting iteration 27.
[2025-11-13 08:14:41,211][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:41,211][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:45,459][__main__][INFO] - Number of regex retries in iteration 27: 0
[2025-11-13 08:14:45,460][__main__][INFO] - agents played in iteration 27 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:14:45,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:45,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:45,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,032][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:46,032][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:46,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:47,088][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:47,412][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:47,735][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:48,060][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:48,385][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:48,709][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:49,033][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:49,356][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:49,680][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:50,003][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:50,326][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:50,650][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:50,974][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:51,298][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:51,623][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:51,947][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:52,270][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:52,593][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:52,916][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:53,239][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:53,562][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:53,885][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:54,207][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:54,530][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:54,855][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:55,179][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:55,502][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:55,826][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:56,149][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:56,472][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:56,795][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:57,125][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:57,860][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:58,631][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:58,632][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:58,634][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:59,649][__main__][INFO] - Iteration 28 took 18s (23.04% Gen, 71.44% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 10m 47s. Estimated total time: 15h 21m 58s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 43s, 500 more iterations: 2h 33m 39s.
[2025-11-13 08:14:59,651][__main__][INFO] - Starting iteration 28.
[2025-11-13 08:14:59,654][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:59,655][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:03,894][__main__][INFO] - Number of regex retries in iteration 28: 0
[2025-11-13 08:15:03,894][__main__][INFO] - agents played in iteration 28 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:15:04,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,451][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:04,452][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:05,210][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:05,507][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:05,832][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:06,157][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:06,485][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:06,808][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:07,132][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:07,455][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:07,782][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:08,104][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:08,428][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:08,750][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:09,074][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:09,399][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:09,722][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:10,046][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:10,370][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:10,696][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:11,025][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:11,347][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:11,670][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:12,003][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:12,327][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:12,651][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:12,974][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:13,298][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:13,621][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:13,944][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:14,267][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:14,590][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:14,913][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:15,237][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:15,562][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:16,287][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:17,039][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:17,040][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:17,042][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:18,051][__main__][INFO] - Iteration 29 took 18s (23.04% Gen, 71.46% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 8m 24s. Estimated total time: 15h 19m 53s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 39s, 500 more iterations: 2h 33m 18s.
[2025-11-13 08:15:18,053][__main__][INFO] - Starting iteration 29.
[2025-11-13 08:15:18,056][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:15:18,057][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:22,310][__main__][INFO] - Number of regex retries in iteration 29: 0
[2025-11-13 08:15:22,311][__main__][INFO] - agents played in iteration 29 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:15:22,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:22,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:22,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:22,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:22,872][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:22,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:23,629][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:23,925][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:24,250][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:24,574][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:24,899][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:25,222][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:25,547][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:25,870][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:26,193][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:26,517][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:26,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:27,165][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:27,490][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:27,822][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:28,147][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:28,471][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:28,794][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:29,116][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:29,440][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:29,764][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:30,087][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:30,411][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:30,736][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:31,060][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:31,383][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:31,706][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:32,030][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:32,353][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:32,679][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:33,008][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:33,332][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:33,657][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:33,980][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:34,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:35,488][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:35,489][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:35,491][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:36,486][__main__][INFO] - Iteration 30 took 18s (23.08% Gen, 71.51% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 9m 44s. Estimated total time: 15h 21m 32s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 43s, 500 more iterations: 2h 33m 35s.
[2025-11-13 08:15:36,488][__main__][INFO] - Starting iteration 30.
[2025-11-13 08:15:36,491][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:15:36,491][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:40,715][__main__][INFO] - Number of regex retries in iteration 30: 0
[2025-11-13 08:15:40,716][__main__][INFO] - agents played in iteration 30 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:15:41,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,258][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,292][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:41,292][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:42,067][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:42,364][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:42,689][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:43,015][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:43,346][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:43,662][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:43,988][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:44,312][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:44,637][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:44,961][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:45,290][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:45,616][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:46,593][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:46,918][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:47,250][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:47,568][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:47,892][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:48,216][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:48,541][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:48,867][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:49,196][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:49,522][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:49,853][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:50,172][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:50,498][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:50,821][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:51,153][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:51,474][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:51,803][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:52,129][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:52,460][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:53,179][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:53,929][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:53,930][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:53,932][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:55,879][__main__][INFO] - Iteration 31 took 19s (21.79% Gen, 68.16% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 57m 20s. Estimated total time: 16h 9m 27s. Time estimates for 10 more iterations: 3m 13s, 100 more iterations: 32m 18s, 500 more iterations: 2h 41m 34s.
[2025-11-13 08:15:55,881][__main__][INFO] - Starting iteration 31.
[2025-11-13 08:15:55,884][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:15:55,884][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:00,814][__main__][INFO] - Number of regex retries in iteration 31: 0
[2025-11-13 08:16:00,815][__main__][INFO] - agents played in iteration 31 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:16:01,278][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,315][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,384][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,385][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:01,385][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:02,160][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:02,455][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:02,780][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:03,106][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:03,433][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:03,756][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:04,079][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:04,404][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:04,727][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:05,051][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:05,379][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:05,702][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:06,026][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:06,354][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:06,677][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:07,000][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:07,323][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:07,646][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:07,970][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:08,294][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:08,618][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:08,940][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:09,264][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:09,586][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:09,909][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:10,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:10,557][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:10,883][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:11,207][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:11,530][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:11,856][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:12,180][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:12,504][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:13,244][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:13,997][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:13,998][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:14,000][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:15,045][__main__][INFO] - Iteration 32 took 19s (25.73% Gen, 68.81% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 45m 40s. Estimated total time: 15h 58m 6s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 56s, 500 more iterations: 2h 39m 41s.
[2025-11-13 08:16:15,047][__main__][INFO] - Starting iteration 32.
[2025-11-13 08:16:15,051][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:15,051][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:19,708][__main__][INFO] - Number of regex retries in iteration 32: 0
[2025-11-13 08:16:19,709][__main__][INFO] - agents played in iteration 32 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:16:20,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,291][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:20,291][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:21,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:21,351][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:21,669][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:21,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:22,323][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:22,647][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:22,975][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:23,298][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:23,623][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:23,951][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:24,273][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:24,596][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:24,923][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:25,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:25,577][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:25,903][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:26,225][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:26,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:26,875][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:27,204][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:27,527][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:27,856][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:28,174][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:28,497][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:28,822][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:29,147][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:29,471][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:29,795][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:30,120][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:30,451][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:30,766][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:31,092][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:31,417][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:32,155][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:32,908][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:32,909][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:32,911][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:33,949][__main__][INFO] - Iteration 33 took 18s (24.64% Gen, 69.86% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 32m 11s. Estimated total time: 15h 44m 56s. Time estimates for 10 more iterations: 3m 8s, 100 more iterations: 31m 29s, 500 more iterations: 2h 37m 29s.
[2025-11-13 08:16:33,951][__main__][INFO] - Starting iteration 33.
[2025-11-13 08:16:33,954][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:33,955][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:38,625][__main__][INFO] - Number of regex retries in iteration 33: 0
[2025-11-13 08:16:38,626][__main__][INFO] - agents played in iteration 33 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:16:39,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:39,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:39,967][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:40,261][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:40,586][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:40,909][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:41,235][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:41,564][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:41,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:42,213][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:42,538][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:42,862][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:43,184][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:43,509][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:43,836][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:44,159][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:44,483][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:44,806][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:45,133][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:45,456][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:45,779][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:46,103][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:46,429][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:46,753][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:47,077][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:47,401][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:47,728][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:48,056][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:48,381][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:48,707][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:49,034][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:49,358][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:49,683][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:50,007][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:50,335][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:51,088][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:51,862][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:51,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:51,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:52,870][__main__][INFO] - Iteration 34 took 18s (24.69% Gen, 69.99% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 32m 44s. Estimated total time: 15h 45m 48s. Time estimates for 10 more iterations: 3m 9s, 100 more iterations: 31m 31s, 500 more iterations: 2h 37m 38s.
[2025-11-13 08:16:52,872][__main__][INFO] - Starting iteration 34.
[2025-11-13 08:16:52,875][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:52,876][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:57,505][__main__][INFO] - Number of regex retries in iteration 34: 0
[2025-11-13 08:16:57,505][__main__][INFO] - agents played in iteration 34 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:16:57,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,100][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,101][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:58,101][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:58,871][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:59,169][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:59,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:59,821][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:00,145][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:00,469][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:00,793][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:01,115][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:01,438][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:01,762][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:02,086][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:02,408][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:02,732][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:03,056][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:03,385][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:03,710][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:04,034][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:04,357][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:04,681][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:05,003][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:05,326][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:05,650][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:05,973][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:06,294][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:06,618][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:06,943][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:07,270][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:07,589][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:07,912][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:08,234][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:08,558][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:08,882][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:09,205][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:09,956][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:10,708][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:10,710][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:10,712][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:11,718][__main__][INFO] - Iteration 35 took 18s (24.57% Gen, 70.08% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 28m 47s. Estimated total time: 15h 42m 10s. Time estimates for 10 more iterations: 3m 8s, 100 more iterations: 31m 24s, 500 more iterations: 2h 37m 1s.
[2025-11-13 08:17:11,720][__main__][INFO] - Starting iteration 35.
[2025-11-13 08:17:11,723][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:11,724][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:16,541][__main__][INFO] - Number of regex retries in iteration 35: 0
[2025-11-13 08:17:16,542][__main__][INFO] - agents played in iteration 35 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:17:17,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,131][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:17,131][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:17,891][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:18,187][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:18,512][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:18,835][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:19,159][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:19,482][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:19,804][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:20,127][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:20,450][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:20,774][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:21,097][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:21,425][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:21,748][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:22,074][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:22,409][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:22,734][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:23,058][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:23,383][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:23,711][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:24,036][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:24,359][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:24,683][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:25,005][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:25,328][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:25,652][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:25,974][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:26,302][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:26,625][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:26,954][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:27,276][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:27,599][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:27,923][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:28,246][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:28,996][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:29,735][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:29,737][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:29,739][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:30,733][__main__][INFO] - Iteration 36 took 19s (25.35% Gen, 69.42% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 36m 50s. Estimated total time: 15h 50m 32s. Time estimates for 10 more iterations: 3m 10s, 100 more iterations: 31m 41s, 500 more iterations: 2h 38m 25s.
[2025-11-13 08:17:30,735][__main__][INFO] - Starting iteration 36.
[2025-11-13 08:17:30,738][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:30,739][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:35,545][__main__][INFO] - Number of regex retries in iteration 36: 0
[2025-11-13 08:17:35,545][__main__][INFO] - agents played in iteration 36 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:17:36,018][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,118][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:36,119][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:36,893][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:37,188][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:37,513][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:37,837][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:38,160][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:38,483][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:38,807][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:39,134][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:39,458][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:39,782][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:40,105][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:40,433][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:40,757][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:41,080][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:41,403][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:41,726][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:42,049][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:42,371][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:42,695][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:43,022][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:43,351][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:43,674][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:43,997][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:44,320][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:44,643][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:44,968][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:45,291][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:45,615][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:45,948][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:46,593][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:46,916][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:47,249][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:47,997][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:48,779][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:48,781][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:48,786][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:49,793][__main__][INFO] - Iteration 37 took 19s (25.22% Gen, 69.48% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 38m 45s. Estimated total time: 15h 52m 46s. Time estimates for 10 more iterations: 3m 10s, 100 more iterations: 31m 45s, 500 more iterations: 2h 38m 47s.
[2025-11-13 08:17:49,795][__main__][INFO] - Starting iteration 37.
[2025-11-13 08:17:49,798][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:49,798][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:54,544][__main__][INFO] - Number of regex retries in iteration 37: 0
[2025-11-13 08:17:54,545][__main__][INFO] - agents played in iteration 37 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:17:55,023][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,127][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:55,127][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:55,890][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:56,186][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:56,510][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:56,835][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:57,160][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:57,487][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:57,809][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:58,138][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:58,462][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:58,785][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:59,110][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:59,434][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:59,757][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:00,084][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:00,409][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:00,732][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:01,056][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:01,385][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:01,706][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:02,029][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:02,353][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:02,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:02,999][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:03,322][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:03,646][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:03,973][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:04,292][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:04,618][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:04,940][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:05,269][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:05,588][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:05,911][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:06,233][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:06,980][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:07,718][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:07,720][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:07,721][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:08,780][__main__][INFO] - Iteration 38 took 18s (25.00% Gen, 69.41% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 34m 48s. Estimated total time: 15h 49m 8s. Time estimates for 10 more iterations: 3m 9s, 100 more iterations: 31m 38s, 500 more iterations: 2h 38m 11s.
[2025-11-13 08:18:08,782][__main__][INFO] - Starting iteration 38.
[2025-11-13 08:18:08,786][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:08,786][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:13,651][__main__][INFO] - Number of regex retries in iteration 38: 0
[2025-11-13 08:18:13,651][__main__][INFO] - agents played in iteration 38 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:18:14,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,225][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,225][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:14,226][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:14,994][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:15,289][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:15,617][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:15,940][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:16,265][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:16,589][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:16,913][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:17,243][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:17,566][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:17,891][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:18,214][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:18,537][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:18,860][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:19,185][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:19,511][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:19,836][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:20,162][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:20,485][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:20,813][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:21,136][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:21,460][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:21,785][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:22,109][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:22,433][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:22,756][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:23,079][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:23,402][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:23,727][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:24,053][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:24,377][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:24,701][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:25,026][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:25,350][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:26,104][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:26,882][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:26,883][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:26,885][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:27,901][__main__][INFO] - Iteration 39 took 19s (25.45% Gen, 69.23% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 41m 10s. Estimated total time: 15h 55m 49s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 51s, 500 more iterations: 2h 39m 18s.
[2025-11-13 08:18:27,903][__main__][INFO] - Starting iteration 39.
[2025-11-13 08:18:27,907][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:27,908][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:32,584][__main__][INFO] - Number of regex retries in iteration 39: 0
[2025-11-13 08:18:32,585][__main__][INFO] - agents played in iteration 39 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:18:33,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:33,176][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:33,923][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:34,222][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:34,545][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:34,872][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:35,201][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:35,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:35,855][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:36,178][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:36,503][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:36,826][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:37,151][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:37,477][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:37,806][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:38,131][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:38,455][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:38,784][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:39,106][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:39,431][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:39,756][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:40,406][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:40,729][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:41,057][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:41,381][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:41,704][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:42,026][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:42,349][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:42,672][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:42,997][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:43,323][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:43,652][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:43,976][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:44,300][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:45,061][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:45,819][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:45,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:45,822][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:46,879][__main__][INFO] - Iteration 40 took 18s (24.65% Gen, 69.77% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 33m 39s. Estimated total time: 15h 48m 37s. Time estimates for 10 more iterations: 3m 9s, 100 more iterations: 31m 37s, 500 more iterations: 2h 38m 6s.
[2025-11-13 08:18:46,881][__main__][INFO] - Starting iteration 40.
[2025-11-13 08:18:46,884][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:46,885][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:51,676][__main__][INFO] - Number of regex retries in iteration 40: 0
[2025-11-13 08:18:51,677][__main__][INFO] - agents played in iteration 40 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:18:52,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,246][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:52,247][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:52,974][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:53,271][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:53,594][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:53,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:54,243][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:54,567][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:54,893][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:55,222][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:55,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:55,867][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:56,193][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:56,519][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:56,845][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:57,169][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:57,494][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:57,818][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:58,141][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:58,464][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:58,789][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:59,114][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:59,437][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:59,760][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:00,083][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:00,407][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:00,730][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:01,052][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:01,375][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:01,706][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:02,024][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:02,350][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:02,676][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:03,005][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:03,322][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:04,063][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:04,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:04,839][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:04,841][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:06,808][__main__][INFO] - Iteration 41 took 19s (24.05% Gen, 66.07% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 20m 56s. Estimated total time: 16h 36m 14s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 12s, 500 more iterations: 2h 46m 2s.
[2025-11-13 08:19:06,811][__main__][INFO] - Starting iteration 41.
[2025-11-13 08:19:06,815][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:06,815][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:12,033][__main__][INFO] - Number of regex retries in iteration 41: 0
[2025-11-13 08:19:12,034][__main__][INFO] - agents played in iteration 41 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:19:12,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,543][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,577][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,612][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:12,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:13,358][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:13,654][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:13,979][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:14,305][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:14,629][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:14,956][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:15,283][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:15,610][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:15,936][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:16,261][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:16,587][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:16,913][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:17,244][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:17,571][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:17,894][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:18,219][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:18,546][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:18,871][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:19,202][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:19,530][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:19,854][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:20,178][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:20,502][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:20,828][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:21,153][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:21,479][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:21,803][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:22,127][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:22,451][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:22,774][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:23,102][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:23,425][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:23,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:24,506][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:25,255][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:25,256][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:25,258][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:26,315][__main__][INFO] - Iteration 42 took 19s (26.76% Gen, 67.81% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 59m 24s. Estimated total time: 16h 15m 1s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 30s, 500 more iterations: 2h 42m 30s.
[2025-11-13 08:19:26,317][__main__][INFO] - Starting iteration 42.
[2025-11-13 08:19:26,320][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:26,321][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:31,285][__main__][INFO] - Number of regex retries in iteration 42: 0
[2025-11-13 08:19:31,286][__main__][INFO] - agents played in iteration 42 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:19:31,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:31,857][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:32,617][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:32,913][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:33,238][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:33,562][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:33,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:34,209][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:34,532][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:34,856][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:35,181][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:35,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:35,827][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:36,150][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:36,476][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:36,799][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:37,125][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:37,449][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:37,772][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:38,096][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:38,419][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:38,742][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:39,065][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:39,390][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:39,714][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:40,036][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:40,360][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:40,683][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:41,006][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:41,334][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:41,657][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:41,979][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:42,304][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:42,627][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:42,951][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:43,699][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:44,452][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:44,454][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:44,456][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:45,430][__main__][INFO] - Iteration 43 took 19s (25.98% Gen, 68.91% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 39m 35s. Estimated total time: 15h 55m 31s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 51s, 500 more iterations: 2h 39m 15s.
[2025-11-13 08:19:45,432][__main__][INFO] - Starting iteration 43.
[2025-11-13 08:19:45,435][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:45,436][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:50,683][__main__][INFO] - Number of regex retries in iteration 43: 0
[2025-11-13 08:19:50,683][__main__][INFO] - agents played in iteration 43 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:19:51,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,254][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:51,254][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:52,018][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:52,314][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:52,964][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:53,297][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:53,620][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:53,943][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:54,266][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:54,601][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:54,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:55,249][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:55,573][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:55,906][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:56,230][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:56,554][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:56,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:57,209][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:57,532][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:57,856][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:58,180][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:58,510][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:58,838][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:59,167][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:59,494][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:59,820][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:00,145][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:00,471][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:00,795][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:01,122][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:01,445][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:01,769][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:02,093][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:02,420][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:03,131][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:03,885][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:03,886][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:03,888][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:04,953][__main__][INFO] - Iteration 44 took 19s (26.88% Gen, 67.65% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 59m 40s. Estimated total time: 16h 15m 56s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 31s, 500 more iterations: 2h 42m 39s.
[2025-11-13 08:20:04,955][__main__][INFO] - Starting iteration 44.
[2025-11-13 08:20:04,959][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:04,960][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:09,951][__main__][INFO] - Number of regex retries in iteration 44: 0
[2025-11-13 08:20:09,952][__main__][INFO] - agents played in iteration 44 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:20:10,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,462][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,531][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:10,531][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:11,283][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:11,588][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:11,913][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:12,242][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:12,567][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:12,895][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:13,221][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:13,545][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:13,871][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:14,197][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:14,523][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:14,854][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:15,180][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:15,505][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:15,830][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:16,156][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:16,486][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:16,817][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:17,140][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:17,467][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:17,790][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:18,114][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:18,437][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:18,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:19,090][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:19,416][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:19,742][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:20,068][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:20,392][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:20,720][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:21,045][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:21,374][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:21,696][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:22,423][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:23,160][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:23,162][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:23,164][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:24,153][__main__][INFO] - Iteration 45 took 19s (26.00% Gen, 68.84% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 43m 8s. Estimated total time: 15h 59m 43s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 59s, 500 more iterations: 2h 39m 57s.
[2025-11-13 08:20:24,155][__main__][INFO] - Starting iteration 45.
[2025-11-13 08:20:24,159][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:24,159][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:29,278][__main__][INFO] - Number of regex retries in iteration 45: 0
[2025-11-13 08:20:29,278][__main__][INFO] - agents played in iteration 45 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:20:29,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,872][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:29,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:30,636][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:30,931][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:31,256][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:31,581][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:31,906][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:32,230][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:32,554][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:32,878][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:33,202][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:33,527][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:33,851][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:34,174][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:34,498][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:34,822][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:35,145][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:35,469][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:35,794][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:36,117][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:36,441][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:36,766][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:37,090][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:37,414][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:37,739][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:38,062][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:38,386][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:38,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:39,038][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:39,359][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:39,682][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:40,006][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:40,335][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:40,655][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:40,978][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:41,717][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:42,456][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:42,458][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:42,459][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:43,439][__main__][INFO] - Iteration 46 took 19s (26.55% Gen, 68.36% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 47m 10s. Estimated total time: 16h 4m 5s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 8s, 500 more iterations: 2h 40m 40s.
[2025-11-13 08:20:43,442][__main__][INFO] - Starting iteration 46.
[2025-11-13 08:20:43,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:43,445][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:48,492][__main__][INFO] - Number of regex retries in iteration 46: 0
[2025-11-13 08:20:48,493][__main__][INFO] - agents played in iteration 46 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:20:48,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:48,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,067][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:49,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:49,844][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:50,140][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:50,464][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:50,790][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:51,117][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:51,448][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:51,774][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:52,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:52,420][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:52,745][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:53,069][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:53,393][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:53,718][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:54,043][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:54,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:54,690][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:55,014][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:55,341][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:55,666][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:55,993][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:56,320][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:56,646][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:56,970][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:57,294][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:57,620][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:57,946][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:58,272][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:58,598][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:58,925][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:59,575][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:59,898][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:00,231][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:00,973][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:01,728][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:01,729][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:01,731][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:02,796][__main__][INFO] - Iteration 47 took 19s (26.08% Gen, 68.41% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 50m 23s. Estimated total time: 16h 7m 37s. Time estimates for 10 more iterations: 3m 13s, 100 more iterations: 32m 15s, 500 more iterations: 2h 41m 16s.
[2025-11-13 08:21:02,798][__main__][INFO] - Starting iteration 47.
[2025-11-13 08:21:02,803][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:02,805][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:07,956][__main__][INFO] - Number of regex retries in iteration 47: 0
[2025-11-13 08:21:07,957][__main__][INFO] - agents played in iteration 47 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:21:08,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,492][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,525][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,525][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:08,526][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:09,267][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:09,562][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:09,887][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:10,212][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:10,537][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:10,862][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:11,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:11,511][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:11,835][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:12,159][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:12,483][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:12,808][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:13,132][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:13,456][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:13,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:14,106][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:14,431][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:14,755][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:15,080][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:15,404][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:15,730][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:16,382][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:16,706][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:17,033][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:17,359][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:17,682][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:18,007][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:18,331][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:18,655][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:18,978][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:19,302][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:19,627][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:20,384][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:21,151][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:21,152][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:21,154][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:22,160][__main__][INFO] - Iteration 48 took 19s (26.61% Gen, 68.17% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 50m 22s. Estimated total time: 16h 7m 55s. Time estimates for 10 more iterations: 3m 13s, 100 more iterations: 32m 15s, 500 more iterations: 2h 41m 19s.
[2025-11-13 08:21:22,161][__main__][INFO] - Starting iteration 48.
[2025-11-13 08:21:22,165][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:22,166][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:27,215][__main__][INFO] - Number of regex retries in iteration 48: 0
[2025-11-13 08:21:27,215][__main__][INFO] - agents played in iteration 48 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:21:27,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,799][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:27,799][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:28,564][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:28,859][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:29,185][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:29,514][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:29,841][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:30,169][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:30,499][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:30,823][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:31,150][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:31,477][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:31,804][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:32,133][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:32,459][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:32,786][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:33,109][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:33,435][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:33,760][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:34,086][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:34,409][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:34,737][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:35,061][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:35,385][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:35,709][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:36,034][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:36,358][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:36,682][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:37,007][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:37,332][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:37,658][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:37,987][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:38,311][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:38,636][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:38,962][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:39,713][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:40,466][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:40,467][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:40,469][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:41,487][__main__][INFO] - Iteration 49 took 19s (26.13% Gen, 68.59% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 48m 16s. Estimated total time: 16h 6m 9s. Time estimates for 10 more iterations: 3m 13s, 100 more iterations: 32m 12s, 500 more iterations: 2h 41m 1s.
[2025-11-13 08:21:41,489][__main__][INFO] - Starting iteration 49.
[2025-11-13 08:21:41,493][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:41,493][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:46,647][__main__][INFO] - Number of regex retries in iteration 49: 0
[2025-11-13 08:21:46,647][__main__][INFO] - agents played in iteration 49 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:21:47,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,229][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:47,229][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:47,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:48,291][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:48,618][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:48,945][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:49,269][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:49,594][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:49,917][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:50,245][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:50,569][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:50,893][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:51,220][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:51,546][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:51,873][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:52,196][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:52,521][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:52,851][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:53,175][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:53,501][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:53,828][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:54,153][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:54,480][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:54,806][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:55,131][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:55,459][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:55,785][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:56,110][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:56,435][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:57,086][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:57,410][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:57,736][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:58,064][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:58,394][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:59,155][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:59,915][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:59,917][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:59,919][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:00,947][__main__][INFO] - Iteration 50 took 19s (26.49% Gen, 68.22% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 54m 31s. Estimated total time: 16h 12m 43s. Time estimates for 10 more iterations: 3m 14s, 100 more iterations: 32m 25s, 500 more iterations: 2h 42m 7s.
[2025-11-13 08:22:00,949][__main__][INFO] - Starting iteration 50.
[2025-11-13 08:22:00,952][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:22:00,952][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:06,054][__main__][INFO] - Number of regex retries in iteration 50: 0
[2025-11-13 08:22:06,054][__main__][INFO] - agents played in iteration 50 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:22:06,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,631][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:06,631][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:07,402][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:07,698][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:08,023][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:08,349][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:08,675][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:09,002][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:09,327][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:09,654][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:09,982][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:10,309][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:10,632][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:10,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:11,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:11,610][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:11,936][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:12,266][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:12,585][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:12,912][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:13,236][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:13,573][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:13,893][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:14,217][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:14,544][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:14,875][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:15,199][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:15,526][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:15,853][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:16,183][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:16,507][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:16,837][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:17,161][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:17,491][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:17,815][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:18,593][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:19,371][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:19,372][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:19,374][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:21,434][__main__][INFO] - Iteration 51 took 20s (24.91% Gen, 65.02% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 45m 37s. Estimated total time: 17h 4m 9s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 8s, 500 more iterations: 2h 50m 41s.
[2025-11-13 08:22:21,437][__main__][INFO] - Starting iteration 51.
[2025-11-13 08:22:21,440][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:22:21,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:26,932][__main__][INFO] - Number of regex retries in iteration 51: 0
[2025-11-13 08:22:26,933][__main__][INFO] - agents played in iteration 51 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:22:27,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,497][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:27,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:28,231][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:28,527][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:28,854][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:29,183][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:29,508][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:29,833][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:30,158][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:30,482][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:30,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:31,130][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:31,457][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:31,784][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:32,108][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:32,437][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:32,763][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:33,088][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:33,411][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:33,736][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:34,065][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:34,394][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:34,719][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:35,046][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:35,370][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:35,696][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:36,020][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:36,344][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:36,669][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:36,994][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:37,319][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:37,644][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:37,969][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:38,294][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:38,618][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:39,384][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:40,155][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:40,156][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:40,158][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:41,153][__main__][INFO] - Iteration 52 took 19s (27.86% Gen, 67.08% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 6m 49s. Estimated total time: 16h 25m 41s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 51s, 500 more iterations: 2h 44m 16s.
[2025-11-13 08:22:41,156][__main__][INFO] - Starting iteration 52.
[2025-11-13 08:22:41,161][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:22:41,162][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:46,485][__main__][INFO] - Number of regex retries in iteration 52: 0
[2025-11-13 08:22:46,486][__main__][INFO] - agents played in iteration 52 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:22:46,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:46,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:47,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:47,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:47,064][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:47,064][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:47,813][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:48,108][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:48,434][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:48,756][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:49,084][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:49,405][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:49,735][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:50,061][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:50,389][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:50,711][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:51,035][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:51,359][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:52,005][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:52,328][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:52,652][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:52,982][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:53,298][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:53,623][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:53,947][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:54,278][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:54,593][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:54,917][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:55,240][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:55,571][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:55,894][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:56,225][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:56,547][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:56,879][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:57,196][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:57,521][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:57,846][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:58,179][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:58,938][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:59,699][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:59,700][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:59,702][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:00,726][__main__][INFO] - Iteration 53 took 19s (27.21% Gen, 67.55% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 59m 5s. Estimated total time: 16h 18m 17s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 36s, 500 more iterations: 2h 43m 2s.
[2025-11-13 08:23:00,728][__main__][INFO] - Starting iteration 53.
[2025-11-13 08:23:00,731][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:00,731][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:06,198][__main__][INFO] - Number of regex retries in iteration 53: 0
[2025-11-13 08:23:06,199][__main__][INFO] - agents played in iteration 53 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:23:06,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,714][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,781][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:06,781][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:07,543][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:07,838][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:08,163][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:08,488][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:08,813][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:09,140][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:09,465][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:09,788][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:10,112][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:10,437][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:10,760][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:11,085][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:11,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:11,738][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:12,061][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:12,386][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:12,712][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:13,036][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:13,362][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:13,686][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:14,010][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:14,335][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:14,659][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:14,984][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:15,307][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:15,632][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:15,958][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:16,282][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:16,605][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:16,930][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:17,254][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:17,583][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:17,907][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:18,666][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:19,440][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:19,442][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:19,444][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:20,429][__main__][INFO] - Iteration 54 took 19s (27.76% Gen, 67.23% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 25s. Estimated total time: 16h 24m 57s. Time estimates for 10 more iterations: 3m 16s, 100 more iterations: 32m 49s, 500 more iterations: 2h 44m 9s.
[2025-11-13 08:23:20,431][__main__][INFO] - Starting iteration 54.
[2025-11-13 08:23:20,435][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:20,436][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:25,825][__main__][INFO] - Number of regex retries in iteration 54: 0
[2025-11-13 08:23:25,826][__main__][INFO] - agents played in iteration 54 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:23:26,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,415][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:26,416][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:27,177][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:27,474][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:27,799][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:28,122][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:28,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:28,770][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:29,093][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:29,416][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:29,740][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:30,066][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:30,390][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:30,715][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:31,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:31,362][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:31,688][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:32,014][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:32,338][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:32,665][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:32,994][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:33,319][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:33,643][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:33,966][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:34,291][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:34,616][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:34,940][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:35,265][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:35,589][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:35,914][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:36,239][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:36,569][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:36,890][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:37,214][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:37,539][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:38,312][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:39,060][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:39,061][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:39,063][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:40,147][__main__][INFO] - Iteration 55 took 19s (27.34% Gen, 67.15% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 46s. Estimated total time: 16h 25m 37s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 51s, 500 more iterations: 2h 44m 16s.
[2025-11-13 08:23:40,149][__main__][INFO] - Starting iteration 55.
[2025-11-13 08:23:40,153][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:40,153][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:45,587][__main__][INFO] - Number of regex retries in iteration 55: 0
[2025-11-13 08:23:45,588][__main__][INFO] - agents played in iteration 55 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:23:46,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,167][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:46,168][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:46,918][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:47,217][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:47,540][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:47,868][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:48,520][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:48,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:49,171][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:49,496][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:49,822][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:50,145][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:50,472][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:50,795][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:51,119][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:51,446][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:51,772][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:52,095][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:52,419][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:52,744][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:53,069][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:53,393][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:53,716][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:54,040][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:54,364][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:54,688][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:55,011][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:55,335][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:55,662][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:55,987][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:56,312][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:56,636][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:56,960][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:57,284][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:58,065][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:58,825][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:58,827][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:58,828][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:59,886][__main__][INFO] - Iteration 56 took 19s (27.54% Gen, 67.09% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 6m 31s. Estimated total time: 16h 26m 42s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 53s, 500 more iterations: 2h 44m 27s.
[2025-11-13 08:23:59,889][__main__][INFO] - Starting iteration 56.
[2025-11-13 08:23:59,892][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:59,893][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:05,265][__main__][INFO] - Number of regex retries in iteration 56: 0
[2025-11-13 08:24:05,266][__main__][INFO] - agents played in iteration 56 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:24:05,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,865][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,865][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:05,866][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:06,618][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:06,914][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:07,239][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:07,563][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:07,888][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:08,213][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:08,545][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:08,869][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:09,195][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:09,520][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:09,844][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:10,169][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:10,494][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:10,819][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:11,146][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:11,473][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:11,798][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:12,122][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:12,447][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:12,772][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:13,098][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:13,422][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:13,749][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:14,072][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:14,397][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:14,721][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:15,045][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:15,370][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:15,695][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:16,020][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:16,349][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:16,680][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:17,009][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:17,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:18,545][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:18,547][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:18,548][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:19,618][__main__][INFO] - Iteration 57 took 19s (27.24% Gen, 67.34% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 48s. Estimated total time: 16h 26m 19s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 52s, 500 more iterations: 2h 44m 23s.
[2025-11-13 08:24:19,620][__main__][INFO] - Starting iteration 57.
[2025-11-13 08:24:19,623][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:19,623][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:24,996][__main__][INFO] - Number of regex retries in iteration 57: 0
[2025-11-13 08:24:24,997][__main__][INFO] - agents played in iteration 57 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:24:25,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:25,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:26,323][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:26,618][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:26,944][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:27,271][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:27,596][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:27,924][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:28,254][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:28,583][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:28,910][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:29,234][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:29,562][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:29,888][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:30,211][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:30,535][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:30,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:31,192][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:31,517][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:31,841][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:32,172][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:32,496][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:32,821][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:33,149][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:33,485][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:33,812][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:34,137][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:34,468][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:34,797][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:35,123][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:35,447][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:35,772][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:36,102][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:36,426][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:36,752][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:37,510][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:38,267][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:38,269][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:38,271][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:39,356][__main__][INFO] - Iteration 58 took 19s (27.23% Gen, 67.26% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 52s. Estimated total time: 16h 26m 42s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 53s, 500 more iterations: 2h 44m 27s.
[2025-11-13 08:24:39,358][__main__][INFO] - Starting iteration 58.
[2025-11-13 08:24:39,362][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:39,362][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:44,699][__main__][INFO] - Number of regex retries in iteration 58: 0
[2025-11-13 08:24:44,700][__main__][INFO] - agents played in iteration 58 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:24:45,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,284][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,285][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:45,285][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:46,040][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:46,340][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:46,665][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:46,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:47,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:47,641][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:47,976][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:48,303][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:48,627][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:48,950][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:49,274][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:49,599][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:49,923][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:50,248][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:50,573][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:50,899][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:51,228][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:51,558][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:51,890][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:52,542][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:52,872][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:53,208][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:53,533][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:53,859][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:54,186][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:54,513][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:54,841][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:55,169][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:55,504][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:55,824][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:56,155][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:56,481][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:57,276][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:58,033][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:58,034][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:58,035][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:59,054][__main__][INFO] - Iteration 59 took 19s (27.10% Gen, 67.72% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 3m 29s. Estimated total time: 16h 24m 39s. Time estimates for 10 more iterations: 3m 16s, 100 more iterations: 32m 49s, 500 more iterations: 2h 44m 6s.
[2025-11-13 08:24:59,056][__main__][INFO] - Starting iteration 59.
[2025-11-13 08:24:59,059][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:59,060][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:04,452][__main__][INFO] - Number of regex retries in iteration 59: 0
[2025-11-13 08:25:04,452][__main__][INFO] - agents played in iteration 59 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:25:04,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:04,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:05,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:05,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:05,037][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:05,038][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:05,777][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:06,071][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:06,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:06,726][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:07,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:07,378][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:07,707][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:08,034][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:08,365][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:08,694][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:09,026][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:09,356][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:09,682][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:10,006][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:10,330][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:10,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:10,986][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:11,316][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:11,639][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:11,964][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:12,288][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:12,613][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:12,937][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:13,262][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:13,586][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:13,914][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:14,238][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:14,567][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:14,896][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:15,223][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:15,549][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:15,873][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:16,198][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:16,952][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:17,723][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:17,725][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:17,726][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:18,973][__main__][INFO] - Iteration 60 took 19s (27.07% Gen, 66.66% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 14m 12s. Estimated total time: 16h 35m 42s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 11s, 500 more iterations: 2h 45m 57s.
[2025-11-13 08:25:18,975][__main__][INFO] - Starting iteration 60.
[2025-11-13 08:25:18,978][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:25:18,979][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:24,269][__main__][INFO] - Number of regex retries in iteration 60: 0
[2025-11-13 08:25:24,270][__main__][INFO] - agents played in iteration 60 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:25:24,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:24,838][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:25,572][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:25,867][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:26,193][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:26,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:26,846][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:27,176][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:27,501][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:27,828][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:28,154][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:28,481][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:28,807][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:29,131][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:29,458][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:29,786][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:30,119][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:30,444][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:30,771][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:31,097][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:31,423][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:31,748][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:32,076][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:32,403][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:32,730][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:33,058][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:33,385][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:33,712][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:34,039][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:34,364][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:34,688][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:35,015][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:35,347][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:35,674][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:35,998][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:36,765][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:37,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:37,529][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:37,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:39,492][__main__][INFO] - Iteration 61 took 20s (25.79% Gen, 64.64% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 43m 53s. Estimated total time: 17h 5m 44s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 11s, 500 more iterations: 2h 50m 57s.
[2025-11-13 08:25:39,494][__main__][INFO] - Starting iteration 61.
[2025-11-13 08:25:39,498][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:25:39,498][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:45,226][__main__][INFO] - Number of regex retries in iteration 61: 0
[2025-11-13 08:25:45,226][__main__][INFO] - agents played in iteration 61 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:25:45,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:45,806][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:46,546][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:46,844][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:47,166][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:47,491][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:47,815][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:48,140][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:48,464][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:48,787][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:49,111][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:49,436][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:49,759][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:50,086][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:50,410][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:50,741][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:51,058][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:51,383][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:51,706][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:52,030][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:52,355][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:52,681][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:53,005][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:53,329][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:53,653][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:53,976][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:54,299][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:54,632][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:54,948][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:55,273][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:55,598][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:55,929][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:56,246][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:56,571][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:56,895][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:57,666][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:58,406][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:58,408][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:58,409][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:59,510][__main__][INFO] - Iteration 62 took 20s (28.62% Gen, 65.87% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 18m 28s. Estimated total time: 16h 40m 39s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 21s, 500 more iterations: 2h 46m 46s.
[2025-11-13 08:25:59,513][__main__][INFO] - Starting iteration 62.
[2025-11-13 08:25:59,516][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:25:59,516][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:05,200][__main__][INFO] - Number of regex retries in iteration 62: 0
[2025-11-13 08:26:05,201][__main__][INFO] - agents played in iteration 62 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:26:05,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,774][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:05,775][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:06,507][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:06,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:07,128][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:07,451][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:07,774][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:08,110][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:08,432][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:08,756][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:09,081][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:09,408][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:09,734][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:10,058][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:10,382][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:10,708][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:11,031][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:11,356][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:11,680][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:12,006][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:12,330][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:12,654][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:12,980][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:13,307][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:13,631][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:13,957][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:14,281][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:14,608][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:14,931][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:15,255][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:15,578][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:15,904][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:16,229][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:16,552][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:16,878][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:17,649][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:18,401][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:18,402][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:18,404][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:19,449][__main__][INFO] - Iteration 63 took 19s (28.52% Gen, 66.23% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 14m 10s. Estimated total time: 16h 36m 40s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 13s, 500 more iterations: 2h 46m 6s.
[2025-11-13 08:26:19,451][__main__][INFO] - Starting iteration 63.
[2025-11-13 08:26:19,455][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:19,455][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:25,124][__main__][INFO] - Number of regex retries in iteration 63: 0
[2025-11-13 08:26:25,124][__main__][INFO] - agents played in iteration 63 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:26:25,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,697][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:25,697][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:26,466][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:26,762][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:27,087][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:27,737][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:28,060][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:28,384][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:28,708][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:29,033][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:29,359][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:29,684][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:30,009][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:30,334][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:30,658][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:30,986][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:31,310][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:31,634][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:31,958][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:32,284][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:32,610][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:32,935][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:33,259][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:33,584][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:33,908][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:34,560][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:34,886][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:35,211][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:35,536][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:35,861][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:36,185][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:36,509][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:36,840][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:37,587][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:38,366][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:38,368][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:38,369][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:39,434][__main__][INFO] - Iteration 64 took 19s (28.37% Gen, 66.29% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 16m 10s. Estimated total time: 16h 39m 1s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 18s, 500 more iterations: 2h 46m 30s.
[2025-11-13 08:26:39,436][__main__][INFO] - Starting iteration 64.
[2025-11-13 08:26:39,442][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:39,442][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:45,078][__main__][INFO] - Number of regex retries in iteration 64: 0
[2025-11-13 08:26:45,079][__main__][INFO] - agents played in iteration 64 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:26:45,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,669][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:45,670][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:46,436][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:46,732][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:47,058][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:47,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:47,715][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:48,045][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:48,375][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:48,702][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:49,026][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:49,353][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:49,677][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:50,001][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:50,324][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:50,647][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:50,971][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:51,295][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:51,619][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:51,944][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:52,267][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:52,590][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:52,917][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:53,243][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:53,567][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:53,896][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:54,228][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:54,552][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:54,877][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:55,201][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:55,527][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:55,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:56,175][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:56,499][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:56,830][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:57,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:58,373][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:58,374][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:58,376][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:59,398][__main__][INFO] - Iteration 65 took 19s (28.24% Gen, 66.63% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 14m 39s. Estimated total time: 16h 37m 50s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 15s, 500 more iterations: 2h 46m 18s.
[2025-11-13 08:26:59,400][__main__][INFO] - Starting iteration 65.
[2025-11-13 08:26:59,403][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:59,404][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:05,040][__main__][INFO] - Number of regex retries in iteration 65: 0
[2025-11-13 08:27:05,040][__main__][INFO] - agents played in iteration 65 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:27:05,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,586][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,620][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:05,620][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:06,396][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:06,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:07,023][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:07,344][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:07,668][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:07,992][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:08,316][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:08,642][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:09,289][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:09,614][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:09,940][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:10,265][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:10,589][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:10,912][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:11,241][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:11,565][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:11,888][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:12,213][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:12,536][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:12,861][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:13,186][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:13,511][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:13,836][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:14,159][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:14,483][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:14,809][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:15,137][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:15,461][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:15,784][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:16,110][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:16,437][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:16,766][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:17,535][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:18,282][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:18,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:18,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:19,288][__main__][INFO] - Iteration 66 took 19s (28.35% Gen, 66.60% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 10m 47s. Estimated total time: 16h 34m 18s. Time estimates for 10 more iterations: 3m 18s, 100 more iterations: 33m 8s, 500 more iterations: 2h 45m 43s.
[2025-11-13 08:27:19,290][__main__][INFO] - Starting iteration 66.
[2025-11-13 08:27:19,294][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:19,295][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:25,011][__main__][INFO] - Number of regex retries in iteration 66: 0
[2025-11-13 08:27:25,012][__main__][INFO] - agents played in iteration 66 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:27:25,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,602][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,602][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:25,602][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:26,375][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:26,671][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:27,999][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:27,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:27,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:27,973][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:28,296][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:28,620][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:28,944][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:29,268][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:29,592][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:29,916][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:30,240][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:30,565][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:30,891][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:31,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:31,552][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:31,882][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:32,208][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:32,538][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:32,862][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:33,186][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:33,510][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:33,840][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:34,165][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:34,490][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:34,814][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:35,465][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:35,790][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:36,115][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:36,443][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:36,767][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:37,537][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:38,287][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:38,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:38,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:39,286][__main__][INFO] - Iteration 67 took 19s (28.60% Gen, 66.42% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 15m 47s. Estimated total time: 16h 39m 38s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 19s, 500 more iterations: 2h 46m 36s.
[2025-11-13 08:27:39,288][__main__][INFO] - Starting iteration 67.
[2025-11-13 08:27:39,292][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:39,292][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:44,971][__main__][INFO] - Number of regex retries in iteration 67: 0
[2025-11-13 08:27:44,972][__main__][INFO] - agents played in iteration 67 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:27:45,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,543][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,544][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:45,544][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:46,295][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:46,590][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:46,922][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:47,247][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:47,573][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:47,898][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:48,234][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:48,558][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:48,882][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:49,206][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:49,543][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:49,868][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:50,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:50,517][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:50,848][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:51,178][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:51,502][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:51,831][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:52,162][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:52,489][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:52,816][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:53,150][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:53,469][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:54,123][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:54,459][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:54,779][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:55,106][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:55,431][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:55,765][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:56,090][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:56,421][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:56,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:57,527][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:58,285][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:58,287][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:58,288][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:59,312][__main__][INFO] - Iteration 68 took 20s (28.36% Gen, 66.51% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 16m 52s. Estimated total time: 16h 41m 3s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 22s, 500 more iterations: 2h 46m 50s.
[2025-11-13 08:27:59,314][__main__][INFO] - Starting iteration 68.
[2025-11-13 08:27:59,317][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:59,318][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:04,927][__main__][INFO] - Number of regex retries in iteration 68: 0
[2025-11-13 08:28:04,928][__main__][INFO] - agents played in iteration 68 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:28:05,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,519][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:05,519][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:06,276][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:06,572][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:06,899][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:07,224][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:07,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:07,874][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:08,200][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:08,527][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:08,852][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:09,177][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:09,501][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:09,827][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:10,151][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:10,475][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:10,803][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:11,129][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:11,455][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:11,782][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:12,107][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:12,432][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:12,758][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:13,084][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:13,410][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:13,733][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:14,058][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:14,383][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:14,709][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:15,034][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:15,361][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:15,688][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:16,012][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:16,338][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:16,662][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:17,431][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:18,188][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:18,189][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:18,191][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:19,192][__main__][INFO] - Iteration 69 took 19s (28.23% Gen, 66.74% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 9m 14s. Estimated total time: 16h 33m 45s. Time estimates for 10 more iterations: 3m 18s, 100 more iterations: 33m 7s, 500 more iterations: 2h 45m 37s.
[2025-11-13 08:28:19,194][__main__][INFO] - Starting iteration 69.
[2025-11-13 08:28:19,197][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:28:19,197][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:24,819][__main__][INFO] - Number of regex retries in iteration 69: 0
[2025-11-13 08:28:24,820][__main__][INFO] - agents played in iteration 69 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:28:25,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,402][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:25,402][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:26,172][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:26,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:26,793][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:27,117][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:27,443][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:27,769][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:28,095][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:28,419][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:28,752][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:29,069][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:29,394][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:29,719][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:30,051][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:30,368][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:30,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:31,018][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:31,347][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:31,668][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:31,996][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:32,320][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:32,652][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:32,973][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:33,297][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:33,622][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:33,954][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:34,273][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:34,597][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:34,922][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:35,247][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:35,573][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:35,898][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:36,223][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:36,554][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:37,303][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:38,060][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:38,062][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:38,063][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:39,069][__main__][INFO] - Iteration 70 took 19s (28.29% Gen, 66.64% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 8m 48s. Estimated total time: 16h 33m 38s. Time estimates for 10 more iterations: 3m 18s, 100 more iterations: 33m 7s, 500 more iterations: 2h 45m 36s.
[2025-11-13 08:28:39,071][__main__][INFO] - Starting iteration 70.
[2025-11-13 08:28:39,075][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:28:39,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:44,738][__main__][INFO] - Number of regex retries in iteration 70: 0
[2025-11-13 08:28:44,738][__main__][INFO] - agents played in iteration 70 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:28:45,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,336][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:45,337][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:46,120][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:46,416][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:46,742][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:47,069][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:47,396][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:47,725][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:48,049][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:48,375][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:48,700][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:49,351][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:49,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:50,000][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:50,325][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:50,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:50,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:51,298][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:51,622][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:51,946][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:52,271][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:52,595][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:52,920][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:53,246][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:53,570][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:54,217][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:54,542][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:54,866][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:55,190][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:55,515][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:55,839][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:56,163][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:56,488][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:57,250][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:58,021][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:58,023][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:58,024][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:59,965][__main__][INFO] - Iteration 71 took 20s (27.11% Gen, 63.59% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 59m 24s. Estimated total time: 17h 24m 35s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 49s, 500 more iterations: 2h 54m 5s.
[2025-11-13 08:28:59,968][__main__][INFO] - Starting iteration 71.
[2025-11-13 08:28:59,971][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:28:59,972][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:06,161][__main__][INFO] - Number of regex retries in iteration 71: 0
[2025-11-13 08:29:06,162][__main__][INFO] - agents played in iteration 71 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:29:06,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,737][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:06,737][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:07,507][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:07,804][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:08,130][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:08,466][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:08,783][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:09,108][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:09,432][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:09,757][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:10,084][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:10,413][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:10,744][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:11,072][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:11,396][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:11,723][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:12,047][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:12,377][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:12,697][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:13,346][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:13,669][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:13,993][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:14,318][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:14,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:14,975][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:15,299][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:15,623][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:15,949][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:16,272][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:16,597][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:16,921][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:17,246][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:17,576][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:17,898][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:18,662][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:29:19,442][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:29:19,443][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:29:19,445][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:20,450][__main__][INFO] - Iteration 72 took 20s (30.22% Gen, 64.86% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 38m 25s. Estimated total time: 17h 3m 57s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 7s, 500 more iterations: 2h 50m 39s.
[2025-11-13 08:29:20,452][__main__][INFO] - Starting iteration 72.
[2025-11-13 08:29:20,456][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:29:20,456][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:26,469][__main__][INFO] - Number of regex retries in iteration 72: 0
[2025-11-13 08:29:26,469][__main__][INFO] - agents played in iteration 72 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:29:26,953][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:26,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,052][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,052][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:27,053][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:27,812][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:28,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:28,433][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:28,758][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:29,085][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:29,408][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:29,732][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:30,057][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:30,381][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:31,354][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:31,678][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:32,002][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:32,330][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:32,653][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:32,976][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:33,301][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:33,625][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:33,950][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:34,274][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:34,598][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:34,922][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:35,248][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:35,572][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:35,898][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:36,226][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:36,550][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:36,875][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:37,199][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:37,523][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:38,173][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:38,960][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:29:39,728][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:29:39,730][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:29:39,732][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:40,740][__main__][INFO] - Iteration 73 took 20s (29.64% Gen, 65.38% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 28m 23s. Estimated total time: 16h 54m 15s. Time estimates for 10 more iterations: 3m 22s, 100 more iterations: 33m 48s, 500 more iterations: 2h 49m 2s.
[2025-11-13 08:29:40,742][__main__][INFO] - Starting iteration 73.
[2025-11-13 08:29:40,746][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:29:40,747][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:46,842][__main__][INFO] - Number of regex retries in iteration 73: 0
[2025-11-13 08:29:46,843][__main__][INFO] - agents played in iteration 73 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:29:47,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,420][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:47,420][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:48,186][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:48,481][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:48,808][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:49,134][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:49,460][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:49,784][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:50,108][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:50,437][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:50,761][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:51,085][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:51,411][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:52,061][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:52,387][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:52,712][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:53,044][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:53,369][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:53,693][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:54,016][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:54,350][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:54,675][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:55,000][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:55,324][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:55,658][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:55,988][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:56,313][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:56,639][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:56,968][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:57,294][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:57,618][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:57,942][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:58,270][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:58,596][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:59,346][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:00,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:00,099][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:00,100][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:01,105][__main__][INFO] - Iteration 74 took 20s (29.94% Gen, 65.12% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 31m 47s. Estimated total time: 16h 57m 59s. Time estimates for 10 more iterations: 3m 23s, 100 more iterations: 33m 55s, 500 more iterations: 2h 49m 39s.
[2025-11-13 08:30:01,107][__main__][INFO] - Starting iteration 74.
[2025-11-13 08:30:01,110][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:01,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:07,035][__main__][INFO] - Number of regex retries in iteration 74: 0
[2025-11-13 08:30:07,035][__main__][INFO] - agents played in iteration 74 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:30:07,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,567][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,601][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,601][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:07,602][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:08,356][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:08,653][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:08,979][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:09,302][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:09,628][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:09,954][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:10,278][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:10,604][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:10,931][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:11,256][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:11,581][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:11,906][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:12,231][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:12,558][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:12,884][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:13,208][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:13,533][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:13,856][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:14,181][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:14,511][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:14,834][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:15,161][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:15,487][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:15,811][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:16,137][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:16,461][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:16,786][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:17,110][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:17,434][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:17,761][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:18,085][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:18,410][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:18,738][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:30:19,497][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:20,250][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:20,252][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:20,253][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:21,266][__main__][INFO] - Iteration 75 took 20s (29.39% Gen, 65.58% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 21m 17s. Estimated total time: 16h 47m 50s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 35s, 500 more iterations: 2h 47m 58s.
[2025-11-13 08:30:21,268][__main__][INFO] - Starting iteration 75.
[2025-11-13 08:30:21,271][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:21,272][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:27,306][__main__][INFO] - Number of regex retries in iteration 75: 0
[2025-11-13 08:30:27,307][__main__][INFO] - agents played in iteration 75 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:30:27,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,880][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:27,880][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:28,647][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:28,944][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:29,269][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:29,596][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:29,920][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:30,244][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:30,569][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:30,897][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:31,226][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:31,552][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:31,878][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:32,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:32,526][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:32,849][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:33,173][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:33,499][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:33,822][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:34,150][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:34,476][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:34,802][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:35,126][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:35,456][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:35,783][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:36,109][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:36,445][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:36,769][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:37,094][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:37,418][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:37,742][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:38,068][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:38,401][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:38,725][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:39,050][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:30:39,790][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:40,562][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:40,563][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:40,565][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:41,590][__main__][INFO] - Iteration 76 took 20s (29.70% Gen, 65.25% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 29m 7s. Estimated total time: 16h 56m 0s. Time estimates for 10 more iterations: 3m 23s, 100 more iterations: 33m 52s, 500 more iterations: 2h 49m 20s.
[2025-11-13 08:30:41,592][__main__][INFO] - Starting iteration 76.
[2025-11-13 08:30:41,595][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:41,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:47,565][__main__][INFO] - Number of regex retries in iteration 76: 0
[2025-11-13 08:30:47,566][__main__][INFO] - agents played in iteration 76 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:30:48,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:48,079][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:48,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:48,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:48,148][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:48,148][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:48,892][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:49,187][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:49,512][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:49,835][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:50,161][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:50,484][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:50,809][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:51,133][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:51,458][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:51,783][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:52,108][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:52,434][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:52,756][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:53,080][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:53,404][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:53,730][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:54,054][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:54,380][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:54,705][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:55,029][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:55,354][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:55,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:56,005][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:56,329][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:56,653][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:56,977][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:57,302][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:57,626][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:57,951][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:58,276][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:58,601][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:58,926][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:59,251][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:31:00,004][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:00,753][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:00,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:00,756][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:01,755][__main__][INFO] - Iteration 77 took 20s (29.61% Gen, 65.43% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 20m 48s. Estimated total time: 16h 48m 1s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 36s, 500 more iterations: 2h 48m 0s.
[2025-11-13 08:31:01,758][__main__][INFO] - Starting iteration 77.
[2025-11-13 08:31:01,762][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:01,762][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:07,700][__main__][INFO] - Number of regex retries in iteration 77: 0
[2025-11-13 08:31:07,701][__main__][INFO] - agents played in iteration 77 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:31:08,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,277][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:08,277][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:09,041][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:09,339][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:09,665][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:09,988][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:10,312][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:10,635][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:10,959][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:11,283][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:11,607][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:11,931][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:12,255][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:12,582][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:12,908][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:13,234][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:13,560][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:13,883][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:14,209][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:14,536][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:14,861][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:15,184][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:15,509][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:15,832][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:16,156][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:16,481][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:16,805][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:17,130][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:17,453][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:17,777][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:18,102][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:18,426][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:18,751][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:19,078][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:19,403][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:31:20,174][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:20,940][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:20,942][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:20,944][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:21,921][__main__][INFO] - Iteration 78 took 20s (29.46% Gen, 65.69% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 20m 27s. Estimated total time: 16h 48m 0s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 36s, 500 more iterations: 2h 48m 0s.
[2025-11-13 08:31:21,923][__main__][INFO] - Starting iteration 78.
[2025-11-13 08:31:21,927][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:21,927][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:27,822][__main__][INFO] - Number of regex retries in iteration 78: 0
[2025-11-13 08:31:27,822][__main__][INFO] - agents played in iteration 78 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:31:28,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,394][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,395][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:28,395][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:29,155][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:29,452][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:29,779][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:30,108][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:30,430][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:30,755][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:31,080][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:31,406][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:31,733][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:32,056][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:32,383][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:32,714][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:33,043][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:33,367][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:33,692][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:34,017][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:34,340][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:34,668][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:34,996][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:35,323][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:35,647][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:35,973][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:36,297][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:36,620][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:36,947][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:37,270][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:37,594][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:37,918][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:38,242][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:38,573][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:39,220][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:39,544][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:31:40,274][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:41,050][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:41,051][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:41,053][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:42,069][__main__][INFO] - Iteration 79 took 20s (29.26% Gen, 65.68% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 19m 16s. Estimated total time: 16h 47m 9s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 34s, 500 more iterations: 2h 47m 51s.
[2025-11-13 08:31:42,071][__main__][INFO] - Starting iteration 79.
[2025-11-13 08:31:42,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:42,074][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:47,972][__main__][INFO] - Number of regex retries in iteration 79: 0
[2025-11-13 08:31:47,973][__main__][INFO] - agents played in iteration 79 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:31:48,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,555][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:48,555][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:49,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:49,614][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:49,941][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:50,266][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:50,592][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:50,918][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:51,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:51,565][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:51,892][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:52,218][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:52,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:52,866][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:53,191][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:53,515][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:53,839][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:54,162][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:54,487][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:54,811][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:55,136][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:55,459][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:55,783][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:56,108][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:56,432][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:56,757][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:57,081][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:57,404][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:57,728][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:58,053][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:58,377][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:58,701][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:59,025][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:59,349][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:59,673][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:00,426][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:01,164][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:01,166][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:01,167][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:02,180][__main__][INFO] - Iteration 80 took 20s (29.34% Gen, 65.62% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 17m 8s. Estimated total time: 16h 45m 21s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 30s, 500 more iterations: 2h 47m 33s.
[2025-11-13 08:32:02,182][__main__][INFO] - Starting iteration 80.
[2025-11-13 08:32:02,186][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:32:02,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:08,077][__main__][INFO] - Number of regex retries in iteration 80: 0
[2025-11-13 08:32:08,078][__main__][INFO] - agents played in iteration 80 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:32:08,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,642][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:08,642][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:09,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:09,699][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:10,025][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:10,351][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:10,676][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:11,000][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:11,324][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:11,648][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:11,972][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:12,298][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:12,622][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:12,950][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:13,274][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:13,598][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:13,922][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:14,246][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:14,572][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:14,895][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:15,219][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:15,543][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:15,867][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:16,191][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:16,514][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:16,837][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:17,160][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:17,484][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:17,808][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:32:18,132][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:32:18,456][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:32:18,781][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:32:19,105][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:32:19,430][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:32:19,754][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:20,482][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:21,225][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:21,226][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:21,228][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:23,423][__main__][INFO] - Iteration 81 took 21s (27.74% Gen, 61.92% Train). Generation: 5s, Training: 13s. Estimated remaining time: 17h 13m 18s. Estimated total time: 17h 41m 52s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 23s, 500 more iterations: 2h 56m 58s.
[2025-11-13 08:32:23,425][__main__][INFO] - Starting iteration 81.
[2025-11-13 08:32:23,428][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:32:23,429][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:29,672][__main__][INFO] - Number of regex retries in iteration 81: 0
[2025-11-13 08:32:29,673][__main__][INFO] - agents played in iteration 81 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:32:30,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,237][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:30,237][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:30,987][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:31,282][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:31,607][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:31,933][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:32,257][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:32,580][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:32,904][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:33,226][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:33,550][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:33,873][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:34,199][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:34,523][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:34,844][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:35,169][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:35,494][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:35,817][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:36,143][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:36,469][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:36,793][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:37,122][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:37,440][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:37,764][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:38,088][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:38,411][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:38,737][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:39,063][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:39,393][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:32:39,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:32:40,050][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:32:40,373][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:32:40,698][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:32:41,023][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:32:41,346][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:42,102][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:42,863][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:42,864][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:42,866][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:43,856][__main__][INFO] - Iteration 82 took 20s (30.56% Gen, 64.58% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 32m 32s. Estimated total time: 17h 1m 27s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 2s, 500 more iterations: 2h 50m 14s.
[2025-11-13 08:32:43,858][__main__][INFO] - Starting iteration 82.
[2025-11-13 08:32:43,862][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:32:43,862][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:50,032][__main__][INFO] - Number of regex retries in iteration 82: 0
[2025-11-13 08:32:50,033][__main__][INFO] - agents played in iteration 82 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:32:50,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,584][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,617][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:50,618][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:51,376][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:51,671][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:51,999][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:52,322][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:52,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:52,974][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:53,307][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:53,633][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:53,959][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:54,285][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:54,610][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:54,936][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:55,266][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:55,597][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:55,924][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:56,256][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:56,580][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:56,908][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:57,237][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:57,564][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:57,893][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:58,221][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:58,544][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:58,869][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:59,193][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:59,521][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:59,842][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:00,167][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:00,491][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:00,820][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:01,140][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:01,466][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:01,791][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:02,575][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:03,338][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:03,339][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:03,341][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:04,381][__main__][INFO] - Iteration 83 took 20s (30.07% Gen, 64.85% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 36m 43s. Estimated total time: 17h 5m 59s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 11s, 500 more iterations: 2h 50m 59s.
[2025-11-13 08:33:04,383][__main__][INFO] - Starting iteration 83.
[2025-11-13 08:33:04,387][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:04,387][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:10,613][__main__][INFO] - Number of regex retries in iteration 83: 0
[2025-11-13 08:33:10,614][__main__][INFO] - agents played in iteration 83 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:33:11,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,186][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:11,186][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:11,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:12,237][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:12,561][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:12,897][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:13,222][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:13,545][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:13,869][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:14,201][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:14,525][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:14,849][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:15,174][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:15,503][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:15,827][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:16,153][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:16,477][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:16,807][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:17,129][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:17,453][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:17,778][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:18,108][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:18,433][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:18,758][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:33:19,083][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:33:19,413][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:33:19,739][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:33:20,062][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:33:20,386][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:20,713][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:21,040][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:21,364][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:21,688][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:22,013][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:22,337][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:23,098][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:23,856][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:23,857][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:23,859][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:24,850][__main__][INFO] - Iteration 84 took 20s (30.43% Gen, 64.72% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 33m 37s. Estimated total time: 17h 3m 13s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 6s, 500 more iterations: 2h 50m 32s.
[2025-11-13 08:33:24,852][__main__][INFO] - Starting iteration 84.
[2025-11-13 08:33:24,856][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:24,856][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:31,110][__main__][INFO] - Number of regex retries in iteration 84: 0
[2025-11-13 08:33:31,110][__main__][INFO] - agents played in iteration 84 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:33:31,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,681][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,682][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:31,682][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:32,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:32,738][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:33,063][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:33,388][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:33,713][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:34,038][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:34,362][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:34,684][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:35,010][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:35,335][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:35,660][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:35,984][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:36,310][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:36,635][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:36,959][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:37,284][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:37,608][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:37,932][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:38,259][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:38,584][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:38,910][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:39,234][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:33:39,558][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:33:39,887][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:33:40,214][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:33:40,539][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:33:40,866][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:41,194][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:41,518][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:41,842][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:42,491][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:42,817][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:43,570][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:44,342][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:44,344][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:44,345][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:45,345][__main__][INFO] - Iteration 85 took 20s (30.52% Gen, 64.60% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 34m 33s. Estimated total time: 17h 4m 30s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 9s, 500 more iterations: 2h 50m 45s.
[2025-11-13 08:33:45,347][__main__][INFO] - Starting iteration 85.
[2025-11-13 08:33:45,350][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:45,350][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:51,566][__main__][INFO] - Number of regex retries in iteration 85: 0
[2025-11-13 08:33:51,566][__main__][INFO] - agents played in iteration 85 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:33:52,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,100][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:52,134][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:52,883][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:53,180][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:53,505][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:53,831][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:54,156][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:54,481][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:54,808][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:55,131][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:55,456][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:55,784][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:56,109][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:56,436][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:56,759][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:57,084][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:57,408][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:57,733][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:58,059][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:58,382][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:58,706][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:59,033][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:59,356][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:59,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:00,004][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:00,329][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:00,654][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:00,978][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:01,302][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:01,626][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:01,952][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:02,278][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:02,605][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:02,935][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:03,265][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:04,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:04,799][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:04,800][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:04,802][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:05,793][__main__][INFO] - Iteration 86 took 20s (30.40% Gen, 64.74% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 31m 56s. Estimated total time: 17h 2m 13s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 4s, 500 more iterations: 2h 50m 22s.
[2025-11-13 08:34:05,795][__main__][INFO] - Starting iteration 86.
[2025-11-13 08:34:05,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:05,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:12,093][__main__][INFO] - Number of regex retries in iteration 86: 0
[2025-11-13 08:34:12,093][__main__][INFO] - agents played in iteration 86 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:34:12,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,697][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:12,697][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:13,479][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:13,774][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:14,103][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:14,427][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:14,751][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:15,076][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:15,400][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:15,724][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:16,049][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:16,373][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:16,697][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:17,022][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:17,346][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:17,670][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:17,996][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:18,320][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:18,644][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:34:18,968][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:34:19,294][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:34:19,622][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:34:19,947][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:34:20,271][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:20,594][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:20,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:21,249][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:21,573][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:21,900][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:22,225][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:22,549][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:22,873][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:23,199][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:23,525][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:23,849][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:24,620][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:25,375][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:25,377][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:25,379][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:26,466][__main__][INFO] - Iteration 87 took 20s (30.45% Gen, 64.28% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 42m 46s. Estimated total time: 17h 13m 24s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 26s, 500 more iterations: 2h 52m 14s.
[2025-11-13 08:34:26,468][__main__][INFO] - Starting iteration 87.
[2025-11-13 08:34:26,472][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:26,472][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:32,735][__main__][INFO] - Number of regex retries in iteration 87: 0
[2025-11-13 08:34:32,736][__main__][INFO] - agents played in iteration 87 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:34:33,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,309][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,310][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:33,311][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:34,088][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:34,383][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:34,710][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:35,038][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:35,357][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:35,681][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:36,006][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:36,337][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:36,656][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:36,984][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:37,309][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:37,634][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:37,959][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:38,285][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:38,610][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:38,935][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:39,261][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:34:39,585][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:34:39,910][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:34:40,235][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:34:40,562][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:34:40,889][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:41,215][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:41,538][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:41,862][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:42,188][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:42,512][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:42,837][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:43,161][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:43,485][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:43,810][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:44,136][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:44,461][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:45,234][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:46,008][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:46,009][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:46,011][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:47,093][__main__][INFO] - Iteration 88 took 20s (30.37% Gen, 64.37% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 40m 9s. Estimated total time: 17h 11m 7s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 22s, 500 more iterations: 2h 51m 51s.
[2025-11-13 08:34:47,095][__main__][INFO] - Starting iteration 88.
[2025-11-13 08:34:47,099][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:47,099][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:53,416][__main__][INFO] - Number of regex retries in iteration 88: 0
[2025-11-13 08:34:53,417][__main__][INFO] - agents played in iteration 88 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:34:53,889][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,925][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,958][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,992][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:53,993][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:54,751][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:55,047][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:55,376][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:55,703][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:56,030][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:56,356][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:56,681][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:57,005][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:57,331][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:57,656][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:57,981][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:58,309][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:58,633][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:58,959][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:59,283][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:59,932][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:00,258][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:00,586][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:00,910][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:01,235][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:01,559][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:01,884][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:02,207][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:02,862][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:03,193][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:03,517][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:03,842][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:04,166][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:04,493][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:04,817][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:05,142][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:05,897][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:06,690][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:06,691][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:06,693][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:07,696][__main__][INFO] - Iteration 89 took 20s (30.67% Gen, 64.45% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 38m 35s. Estimated total time: 17h 9m 53s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 19s, 500 more iterations: 2h 51m 38s.
[2025-11-13 08:35:07,698][__main__][INFO] - Starting iteration 89.
[2025-11-13 08:35:07,701][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:35:07,702][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:13,995][__main__][INFO] - Number of regex retries in iteration 89: 0
[2025-11-13 08:35:13,995][__main__][INFO] - agents played in iteration 89 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:35:14,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,561][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:14,562][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:15,310][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:15,605][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:15,930][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:16,255][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:16,579][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:16,905][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:17,235][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:35:17,559][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:35:17,884][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:35:18,209][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:35:18,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:35:18,860][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:35:19,185][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:35:19,511][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:35:19,841][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:35:20,161][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:35:20,486][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:20,811][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:21,136][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:21,461][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:21,786][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:22,112][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:22,439][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:22,763][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:23,088][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:23,413][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:23,738][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:24,065][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:24,391][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:24,716][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:25,040][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:25,364][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:25,689][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:26,482][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:27,245][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:27,246][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:27,248][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:28,288][__main__][INFO] - Iteration 90 took 20s (30.57% Gen, 64.37% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 37m 42s. Estimated total time: 17h 9m 21s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 18s, 500 more iterations: 2h 51m 33s.
[2025-11-13 08:35:28,290][__main__][INFO] - Starting iteration 90.
[2025-11-13 08:35:28,295][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:35:28,295][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:34,505][__main__][INFO] - Number of regex retries in iteration 90: 0
[2025-11-13 08:35:34,506][__main__][INFO] - agents played in iteration 90 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:35:34,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,008][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,074][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,075][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:35,075][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:35,840][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:36,136][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:36,462][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:36,787][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:37,111][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:37,437][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:37,762][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:35:38,086][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:35:38,412][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:35:38,736][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:35:39,062][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:35:39,387][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:35:39,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:35:40,037][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:35:40,364][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:35:40,691][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:35:41,023][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:41,349][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:41,673][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:41,998][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:42,328][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:42,656][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:42,982][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:43,310][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:43,644][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:43,975][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:44,303][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:44,630][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:44,955][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:45,280][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:45,606][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:45,931][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:46,256][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:47,035][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:47,791][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:47,792][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:47,794][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:49,802][__main__][INFO] - Iteration 91 took 21s (28.88% Gen, 61.78% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 23m 23s. Estimated total time: 17h 55m 24s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 50s, 500 more iterations: 2h 59m 14s.
[2025-11-13 08:35:49,804][__main__][INFO] - Starting iteration 91.
[2025-11-13 08:35:49,807][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:35:49,807][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:56,314][__main__][INFO] - Number of regex retries in iteration 91: 0
[2025-11-13 08:35:56,315][__main__][INFO] - agents played in iteration 91 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:35:56,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:56,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:56,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:56,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:56,899][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:56,900][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:57,670][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:57,965][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:58,290][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:58,615][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:58,940][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:59,266][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:59,594][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:35:59,919][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:00,243][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:00,567][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:00,892][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:01,216][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:01,540][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:01,865][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:02,188][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:02,513][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:02,837][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:03,163][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:03,487][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:03,812][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:04,136][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:04,460][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:04,786][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:05,112][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:05,437][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:05,764][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:06,089][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:06,413][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:06,737][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:07,061][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:07,385][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:07,710][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:08,034][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:08,812][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:09,556][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:09,557][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:09,559][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:10,765][__main__][INFO] - Iteration 92 took 20s (31.05% Gen, 63.19% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 55m 33s. Estimated total time: 17h 27m 55s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 55s, 500 more iterations: 2h 54m 39s.
[2025-11-13 08:36:10,767][__main__][INFO] - Starting iteration 92.
[2025-11-13 08:36:10,770][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:10,770][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:17,165][__main__][INFO] - Number of regex retries in iteration 92: 0
[2025-11-13 08:36:17,166][__main__][INFO] - agents played in iteration 92 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:36:17,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,738][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:17,738][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:36:18,501][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:36:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:36:19,122][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:36:19,446][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:36:19,770][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:36:20,094][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:36:20,421][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:36:20,746][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:21,071][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:21,397][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:21,724][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:22,054][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:22,379][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:22,707][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:23,033][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:23,362][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:23,683][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:24,008][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:24,332][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:24,657][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:24,984][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:25,312][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:25,642][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:25,970][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:26,297][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:26,623][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:26,950][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:27,599][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:27,925][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:28,251][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:28,576][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:28,902][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:29,657][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:30,431][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:30,432][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:30,434][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:31,481][__main__][INFO] - Iteration 93 took 20s (30.88% Gen, 64.06% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 42m 52s. Estimated total time: 17h 15m 35s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 31s, 500 more iterations: 2h 52m 35s.
[2025-11-13 08:36:31,483][__main__][INFO] - Starting iteration 93.
[2025-11-13 08:36:31,485][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:31,486][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:38,025][__main__][INFO] - Number of regex retries in iteration 93: 0
[2025-11-13 08:36:38,026][__main__][INFO] - agents played in iteration 93 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:36:38,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,595][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:38,595][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:36:39,357][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:36:39,653][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:36:39,979][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:36:40,305][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:36:40,635][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:36:40,963][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:36:41,292][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:36:41,622][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:41,949][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:42,275][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:42,600][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:42,926][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:43,250][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:43,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:43,898][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:44,224][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:44,551][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:44,877][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:45,202][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:45,526][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:45,851][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:46,176][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:46,502][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:46,826][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:47,152][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:47,476][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:48,126][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:48,451][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:48,775][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:49,100][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:49,425][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:49,749][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:50,501][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:51,303][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:51,304][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:51,305][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:52,344][__main__][INFO] - Iteration 94 took 20s (31.35% Gen, 63.66% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 49m 53s. Estimated total time: 17h 22m 56s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 45s, 500 more iterations: 2h 53m 49s.
[2025-11-13 08:36:52,346][__main__][INFO] - Starting iteration 94.
[2025-11-13 08:36:52,348][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:52,349][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:58,875][__main__][INFO] - Number of regex retries in iteration 94: 0
[2025-11-13 08:36:58,875][__main__][INFO] - agents played in iteration 94 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:36:59,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,384][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,450][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:59,450][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:00,216][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:00,512][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:00,837][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:01,167][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:01,486][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:01,811][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:02,136][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:02,467][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:02,788][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:03,113][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:03,437][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:03,762][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:04,086][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:04,412][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:04,736][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:05,062][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:05,385][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:05,709][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:06,034][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:06,359][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:06,686][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:07,009][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:07,334][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:07,664][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:07,983][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:08,308][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:08,633][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:08,963][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:09,281][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:09,605][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:09,932][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:10,256][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:10,581][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:11,364][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:12,137][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:12,138][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:12,140][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:13,168][__main__][INFO] - Iteration 95 took 20s (31.35% Gen, 63.71% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 47m 36s. Estimated total time: 17h 21m 0s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 42s, 500 more iterations: 2h 53m 30s.
[2025-11-13 08:37:13,170][__main__][INFO] - Starting iteration 95.
[2025-11-13 08:37:13,174][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:13,174][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:37:19,744][__main__][INFO] - Number of regex retries in iteration 95: 0
[2025-11-13 08:37:19,744][__main__][INFO] - agents played in iteration 95 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:37:20,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,305][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:37:20,305][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:21,086][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:21,380][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:21,706][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:22,035][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:22,359][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:22,684][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:23,009][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:23,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:23,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:23,987][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:24,312][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:24,638][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:24,963][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:25,288][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:25,612][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:25,947][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:26,273][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:26,598][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:26,923][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:27,253][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:27,578][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:27,903][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:28,228][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:28,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:28,877][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:29,199][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:29,524][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:29,848][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:30,173][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:30,497][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:30,822][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:31,153][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:31,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:32,269][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:33,023][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:33,025][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:33,027][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:34,055][__main__][INFO] - Iteration 96 took 20s (31.47% Gen, 63.61% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 50m 19s. Estimated total time: 17h 24m 4s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 48s, 500 more iterations: 2h 54m 0s.
[2025-11-13 08:37:34,057][__main__][INFO] - Starting iteration 96.
[2025-11-13 08:37:34,060][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:34,061][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:37:40,605][__main__][INFO] - Number of regex retries in iteration 96: 0
[2025-11-13 08:37:40,605][__main__][INFO] - agents played in iteration 96 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:37:41,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:41,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:41,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:41,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:41,181][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:37:41,181][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:41,952][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:42,248][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:42,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:42,896][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:43,221][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:43,545][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:43,876][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:44,196][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:44,521][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:44,847][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:45,172][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:45,497][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:45,823][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:46,147][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:46,472][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:47,121][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:47,446][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:47,771][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:48,095][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:48,421][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:48,745][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:49,075][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:49,719][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:50,043][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:50,372][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:50,692][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:51,017][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:51,340][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:51,671][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:51,990][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:52,315][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:53,093][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:53,847][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:53,849][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:53,850][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:54,897][__main__][INFO] - Iteration 97 took 20s (31.40% Gen, 63.56% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 47m 49s. Estimated total time: 17h 21m 55s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 43s, 500 more iterations: 2h 53m 39s.
[2025-11-13 08:37:54,900][__main__][INFO] - Starting iteration 97.
[2025-11-13 08:37:54,903][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:54,904][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:01,372][__main__][INFO] - Number of regex retries in iteration 97: 0
[2025-11-13 08:38:01,373][__main__][INFO] - agents played in iteration 97 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:38:01,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,946][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:01,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:02,707][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:03,003][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:03,328][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:03,659][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:03,979][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:04,303][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:04,627][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:04,952][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:05,278][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:05,603][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:05,930][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:06,255][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:06,584][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:06,908][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:07,562][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:07,889][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:08,213][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:08,538][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:08,865][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:09,190][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:09,513][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:09,837][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:10,164][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:10,488][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:10,814][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:11,142][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:11,472][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:11,799][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:12,123][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:12,451][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:12,775][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:13,102][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:13,863][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:14,624][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:14,625][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:14,627][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:15,596][__main__][INFO] - Iteration 98 took 20s (31.26% Gen, 64.05% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 40m 14s. Estimated total time: 17h 14m 41s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 29s, 500 more iterations: 2h 52m 26s.
[2025-11-13 08:38:15,598][__main__][INFO] - Starting iteration 98.
[2025-11-13 08:38:15,601][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:15,602][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:22,055][__main__][INFO] - Number of regex retries in iteration 98: 0
[2025-11-13 08:38:22,056][__main__][INFO] - agents played in iteration 98 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:38:22,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,561][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,629][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:22,629][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:23,395][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:23,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:24,021][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:24,347][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:24,672][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:24,999][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:25,325][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:25,648][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:25,973][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:26,296][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:26,620][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:26,947][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:27,600][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:27,924][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:28,248][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:28,572][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:28,896][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:29,220][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:29,544][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:29,870][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:30,194][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:30,519][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:30,846][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:31,170][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:31,494][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:31,819][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:32,143][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:32,467][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:32,792][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:33,117][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:33,443][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:33,767][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:34,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:35,293][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:35,294][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:35,296][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:36,317][__main__][INFO] - Iteration 99 took 20s (31.15% Gen, 63.91% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 41m 3s. Estimated total time: 17h 15m 51s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 31s, 500 more iterations: 2h 52m 38s.
[2025-11-13 08:38:36,320][__main__][INFO] - Starting iteration 99.
[2025-11-13 08:38:36,323][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:36,323][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:42,903][__main__][INFO] - Number of regex retries in iteration 99: 0
[2025-11-13 08:38:42,904][__main__][INFO] - agents played in iteration 99 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:38:43,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,448][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,481][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,482][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:43,482][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:44,227][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:44,523][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:44,848][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:45,175][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:45,499][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:45,824][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:46,147][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:46,471][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:47,122][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:47,451][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:47,782][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:48,108][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:48,440][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:48,766][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:49,089][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:49,416][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:49,748][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:50,073][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:50,397][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:50,722][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:51,057][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:51,381][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:51,708][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:52,031][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:52,357][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:52,684][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:53,008][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:53,334][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:53,664][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:53,989][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:54,312][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:54,638][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:55,384][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:56,149][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:56,151][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:56,152][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:57,164][__main__][INFO] - Iteration 100 took 20s (31.57% Gen, 63.57% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 46m 57s. Estimated total time: 17h 22m 6s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 44s, 500 more iterations: 2h 53m 41s.
[2025-11-13 08:38:57,166][__main__][INFO] - Starting iteration 100.
[2025-11-13 08:38:57,169][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:57,169][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:03,675][__main__][INFO] - Number of regex retries in iteration 100: 0
[2025-11-13 08:39:03,676][__main__][INFO] - agents played in iteration 100 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:39:04,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,243][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:04,244][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:05,014][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:05,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:05,636][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:05,962][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:06,290][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:06,616][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:06,941][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:07,266][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:07,593][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:07,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:08,247][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:08,571][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:08,896][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:09,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:09,545][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:09,871][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:10,199][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:10,525][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:10,853][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:11,177][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:11,502][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:11,827][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:12,156][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:12,486][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:12,813][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:13,138][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:13,466][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:13,796][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:14,120][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:14,445][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:14,769][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:15,094][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:15,420][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:16,189][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:39:16,943][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:39:16,945][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:39:16,946][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:39:18,899][__main__][INFO] - Iteration 101 took 21s (29.94% Gen, 61.07% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 31m 1s. Estimated total time: 18h 6m 31s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 13s, 500 more iterations: 3h 1m 5s.
[2025-11-13 08:39:18,901][__main__][INFO] - Starting iteration 101.
[2025-11-13 08:39:18,904][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:39:18,905][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:25,797][__main__][INFO] - Number of regex retries in iteration 101: 0
[2025-11-13 08:39:25,797][__main__][INFO] - agents played in iteration 101 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:39:26,272][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,305][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,372][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:26,372][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:27,144][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:27,439][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:27,765][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:28,091][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:28,416][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:28,741][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:29,066][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:29,390][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:29,714][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:30,038][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:30,364][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:30,688][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:31,012][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:31,336][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:31,664][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:31,988][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:32,313][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:32,636][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:32,963][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:33,288][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:33,613][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:33,937][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:34,261][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:34,585][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:34,909][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:35,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:35,557][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:35,883][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:36,206][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:36,856][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:37,180][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:37,505][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:38,274][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:39:39,034][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:39:39,036][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:39:39,037][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:39:40,074][__main__][INFO] - Iteration 102 took 21s (32.56% Gen, 62.54% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 2m 39s. Estimated total time: 17h 38m 30s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 17s, 500 more iterations: 2h 56m 25s.
[2025-11-13 08:39:40,076][__main__][INFO] - Starting iteration 102.
[2025-11-13 08:39:40,079][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:39:40,080][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:46,941][__main__][INFO] - Number of regex retries in iteration 102: 0
[2025-11-13 08:39:46,942][__main__][INFO] - agents played in iteration 102 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:39:47,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,521][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:47,521][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:48,257][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:48,551][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:48,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:49,205][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:49,534][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:49,859][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:50,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:50,509][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:50,834][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:51,161][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:51,486][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:51,816][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:52,138][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:52,463][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:52,788][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:53,113][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:53,438][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:53,761][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:54,086][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:54,411][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:54,735][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:55,059][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:55,384][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:55,708][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:56,034][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:56,359][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:56,689][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:57,007][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:57,331][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:57,656][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:57,980][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:58,304][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:58,629][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:59,393][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:00,159][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:00,161][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:00,163][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:01,164][__main__][INFO] - Iteration 103 took 21s (32.54% Gen, 62.70% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 58m 3s. Estimated total time: 17h 34m 15s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 8s, 500 more iterations: 2h 55m 42s.
[2025-11-13 08:40:01,166][__main__][INFO] - Starting iteration 103.
[2025-11-13 08:40:01,169][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:01,169][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:07,998][__main__][INFO] - Number of regex retries in iteration 103: 0
[2025-11-13 08:40:07,999][__main__][INFO] - agents played in iteration 103 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:40:08,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,574][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:08,574][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:09,344][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:09,640][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:09,967][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:10,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:10,942][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:11,270][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:11,594][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:11,919][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:12,244][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:12,571][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:12,896][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:13,221][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:13,546][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:13,874][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:14,200][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:14,525][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:14,849][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:15,173][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:15,499][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:15,824][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:16,148][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:16,477][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:16,797][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:40:17,122][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:40:17,448][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:40:17,778][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:40:18,099][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:40:18,424][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:40:18,749][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:40:19,073][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:40:19,399][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:40:19,723][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:40:20,524][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:21,282][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:21,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:21,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:22,310][__main__][INFO] - Iteration 104 took 21s (32.30% Gen, 62.84% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 0m 33s. Estimated total time: 17h 37m 7s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 14s, 500 more iterations: 2h 56m 11s.
[2025-11-13 08:40:22,312][__main__][INFO] - Starting iteration 104.
[2025-11-13 08:40:22,316][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:22,317][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:25,600][mllm.models.large_language_model_local][WARNING] - Response <IbrahimStartingMove> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 08:40:30,282][__main__][INFO] - Number of regex retries in iteration 104: 1
[2025-11-13 08:40:30,283][__main__][INFO] - agents played in iteration 104 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:40:30,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:30,810][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:30,843][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:30,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:30,877][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:30,877][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:31,558][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:31,853][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:32,181][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:32,506][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:32,831][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:33,155][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:33,481][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:33,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:34,130][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:34,456][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:34,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:35,106][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:35,429][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:35,754][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:36,078][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:36,404][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:36,731][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:37,055][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:37,380][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:37,705][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:38,030][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:38,355][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:38,681][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:39,005][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:40:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:40:39,654][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:40:39,978][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:40:40,304][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:40:40,631][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:40:40,958][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:40:41,289][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:40:41,612][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:40:41,938][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:40:42,714][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:43,452][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:43,455][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:43,456][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:44,341][__main__][INFO] - Iteration 105 took 22s (36.17% Gen, 59.81% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 44m 21s. Estimated total time: 18h 21m 16s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 42s, 500 more iterations: 3h 3m 32s.
[2025-11-13 08:40:44,343][__main__][INFO] - Starting iteration 105.
[2025-11-13 08:40:44,346][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:44,346][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:50,927][__main__][INFO] - Number of regex retries in iteration 105: 0
[2025-11-13 08:40:50,928][__main__][INFO] - agents played in iteration 105 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:40:51,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:51,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:51,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:51,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:51,537][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:51,537][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:52,234][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:52,531][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:52,856][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:53,182][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:53,511][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:53,834][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:54,158][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:54,485][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:54,816][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:55,143][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:55,470][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:55,799][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:56,123][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:56,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:56,772][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:57,097][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:57,421][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:57,749][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:58,075][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:58,401][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:58,731][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:59,062][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:59,390][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:59,719][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:41:00,041][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:41:00,368][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:41:00,694][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:01,023][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:01,350][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:01,677][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:02,001][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:02,329][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:02,653][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:03,426][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:04,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:04,193][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:04,194][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:05,135][__main__][INFO] - Iteration 106 took 20s (31.66% Gen, 63.81% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 42m 15s. Estimated total time: 17h 19m 32s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 39s, 500 more iterations: 2h 53m 15s.
[2025-11-13 08:41:05,137][__main__][INFO] - Starting iteration 106.
[2025-11-13 08:41:05,140][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:05,141][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:11,641][__main__][INFO] - Number of regex retries in iteration 106: 0
[2025-11-13 08:41:11,641][__main__][INFO] - agents played in iteration 106 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:41:12,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:12,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:12,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:12,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:12,237][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:12,237][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:12,935][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:13,230][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:13,554][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:13,879][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:14,207][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:14,531][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:14,862][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:15,190][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:15,521][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:15,839][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:16,165][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:16,490][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:16,814][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:17,138][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:17,462][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:17,788][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:41:18,113][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:41:18,442][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:41:18,769][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:41:19,094][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:41:19,418][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:41:19,742][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:41:20,068][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:41:20,393][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:41:20,722][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:41:21,048][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:41:21,374][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:21,699][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:22,025][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:22,349][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:22,676][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:23,002][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:23,327][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:24,068][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:24,818][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:24,819][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:24,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:25,764][__main__][INFO] - Iteration 107 took 20s (31.52% Gen, 63.90% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 33m 36s. Estimated total time: 17h 11m 12s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 22s, 500 more iterations: 2h 51m 52s.
[2025-11-13 08:41:25,765][__main__][INFO] - Starting iteration 107.
[2025-11-13 08:41:25,768][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:25,769][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:32,474][__main__][INFO] - Number of regex retries in iteration 107: 0
[2025-11-13 08:41:32,474][__main__][INFO] - agents played in iteration 107 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:41:32,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:32,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:33,028][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:33,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:33,063][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:33,064][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:33,801][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:34,096][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:34,423][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:34,748][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:35,078][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:35,402][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:35,731][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:36,054][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:36,378][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:36,704][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:37,028][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:37,353][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:37,678][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:38,004][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:38,329][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:38,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:41:38,979][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:41:39,308][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:41:39,632][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:41:39,959][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:41:40,285][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:41:40,613][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:41:40,937][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:41:41,263][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:41:41,590][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:41:41,917][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:41:42,245][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:42,572][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:42,896][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:43,221][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:43,548][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:43,877][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:44,205][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:45,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:45,881][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:45,883][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:45,885][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:46,847][__main__][INFO] - Iteration 108 took 21s (31.81% Gen, 63.62% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 56m 1s. Estimated total time: 17h 33m 59s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 7s, 500 more iterations: 2h 55m 39s.
[2025-11-13 08:41:46,849][__main__][INFO] - Starting iteration 108.
[2025-11-13 08:41:46,852][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:46,852][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:53,501][__main__][INFO] - Number of regex retries in iteration 108: 0
[2025-11-13 08:41:53,502][__main__][INFO] - agents played in iteration 108 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:41:53,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:54,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:54,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:54,078][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:54,078][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:54,079][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:54,829][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:55,129][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:55,448][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:55,775][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:56,103][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:56,434][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:56,764][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:57,090][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:57,416][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:57,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:58,077][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:58,404][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:58,734][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:59,065][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:59,389][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:59,717][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:42:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:00,372][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:00,699][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:01,026][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:01,355][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:01,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:02,006][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:02,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:02,664][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:02,991][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:03,321][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:03,648][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:03,972][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:04,297][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:04,624][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:04,953][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:05,281][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:06,033][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:06,776][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:06,778][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:06,779][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:07,751][__main__][INFO] - Iteration 109 took 20s (31.82% Gen, 63.53% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 46m 41s. Estimated total time: 17h 25m 0s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 50s, 500 more iterations: 2h 54m 10s.
[2025-11-13 08:42:07,753][__main__][INFO] - Starting iteration 109.
[2025-11-13 08:42:07,755][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:42:07,756][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:14,539][__main__][INFO] - Number of regex retries in iteration 109: 0
[2025-11-13 08:42:14,539][__main__][INFO] - agents played in iteration 109 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:42:15,049][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:15,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:15,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:15,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:15,154][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:15,154][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:15,907][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:16,201][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:16,527][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:42:16,857][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:42:17,184][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:42:17,510][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:42:17,834][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:42:18,159][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:42:18,485][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:42:18,813][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:42:19,138][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:42:19,464][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:42:19,789][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:42:20,116][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:42:20,441][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:42:20,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:42:21,089][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:21,417][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:21,740][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:22,067][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:22,394][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:22,722][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:23,048][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:23,376][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:23,700][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:24,024][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:24,348][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:24,673][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:24,998][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:25,322][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:25,647][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:25,971][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:26,296][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:27,062][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:27,818][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:27,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:27,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:28,783][__main__][INFO] - Iteration 110 took 21s (32.26% Gen, 63.16% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 52m 46s. Estimated total time: 17h 31m 26s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 2s, 500 more iterations: 2h 55m 14s.
[2025-11-13 08:42:28,785][__main__][INFO] - Starting iteration 110.
[2025-11-13 08:42:28,788][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:42:28,788][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:35,507][__main__][INFO] - Number of regex retries in iteration 110: 0
[2025-11-13 08:42:35,507][__main__][INFO] - agents played in iteration 110 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:42:35,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:36,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:36,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:36,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:36,098][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:36,099][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:36,871][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:37,168][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:42:37,819][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:42:38,143][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:42:38,469][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:42:38,797][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:42:39,126][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:42:39,451][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:42:39,778][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:42:40,105][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:42:40,430][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:42:40,757][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:42:41,081][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:42:41,406][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:42:41,731][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:42:42,057][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:42,382][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:42,712][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:43,036][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:43,362][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:43,690][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:44,017][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:44,344][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:44,669][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:44,996][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:45,322][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:45,649][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:45,974][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:46,302][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:46,630][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:46,958][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:47,284][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:48,052][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:48,789][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:48,790][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:48,792][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:50,768][__main__][INFO] - Iteration 111 took 21s (30.56% Gen, 60.44% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 40m 1s. Estimated total time: 18h 19m 3s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 38s, 500 more iterations: 3h 3m 10s.
[2025-11-13 08:42:50,770][__main__][INFO] - Starting iteration 111.
[2025-11-13 08:42:50,772][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:42:50,773][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:57,907][__main__][INFO] - Number of regex retries in iteration 111: 0
[2025-11-13 08:42:57,908][__main__][INFO] - agents played in iteration 111 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:42:58,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,501][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:58,501][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:59,272][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:59,567][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:59,892][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:43:00,216][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:00,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:00,866][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:01,190][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:01,515][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:01,843][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:02,171][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:02,496][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:02,822][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:03,149][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:03,474][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:03,799][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:04,125][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:04,450][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:04,776][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:05,101][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:05,426][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:05,752][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:06,076][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:06,401][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:06,726][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:07,051][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:07,376][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:07,700][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:08,350][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:08,675][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:09,002][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:09,326][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:09,652][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:10,426][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:11,176][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:11,178][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:11,180][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:12,158][__main__][INFO] - Iteration 112 took 21s (33.36% Gen, 62.06% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 9m 56s. Estimated total time: 17h 49m 20s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 38s, 500 more iterations: 2h 58m 13s.
[2025-11-13 08:43:12,161][__main__][INFO] - Starting iteration 112.
[2025-11-13 08:43:12,163][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:12,164][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:43:19,031][__main__][INFO] - Number of regex retries in iteration 112: 0
[2025-11-13 08:43:19,032][__main__][INFO] - agents played in iteration 112 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:43:19,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,625][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:43:19,625][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:43:20,403][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:43:20,699][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:43:21,026][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:43:21,353][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:21,681][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:22,009][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:22,340][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:22,667][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:22,994][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:23,320][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:23,646][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:23,973][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:24,299][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:24,627][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:24,952][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:25,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:25,607][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:25,935][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:26,260][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:26,585][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:26,910][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:27,239][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:27,564][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:27,888][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:28,216][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:28,548][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:28,873][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:29,198][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:29,521][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:29,847][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:30,171][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:30,496][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:30,821][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:31,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:32,341][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:32,343][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:32,344][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:33,361][__main__][INFO] - Iteration 113 took 21s (32.40% Gen, 62.80% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 0m 11s. Estimated total time: 17h 39m 55s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 19s, 500 more iterations: 2h 56m 39s.
[2025-11-13 08:43:33,364][__main__][INFO] - Starting iteration 113.
[2025-11-13 08:43:33,367][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:33,367][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:43:40,348][__main__][INFO] - Number of regex retries in iteration 113: 0
[2025-11-13 08:43:40,349][__main__][INFO] - agents played in iteration 113 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:43:40,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,870][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,904][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,939][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:43:40,939][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:43:41,713][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:43:42,009][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:43:42,334][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:43:42,663][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:42,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:43,312][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:43,637][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:43,963][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:44,289][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:44,611][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:44,935][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:45,262][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:45,586][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:45,911][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:46,235][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:46,559][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:46,884][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:47,209][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:47,533][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:47,857][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:48,182][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:48,505][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:48,830][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:49,154][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:49,479][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:49,803][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:50,128][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:50,453][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:50,779][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:51,104][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:51,428][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:51,753][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:52,078][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:52,839][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:53,613][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:53,614][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:53,616][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:54,605][__main__][INFO] - Iteration 114 took 21s (32.87% Gen, 62.47% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 1m 52s. Estimated total time: 17h 41m 57s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 23s, 500 more iterations: 2h 56m 59s.
[2025-11-13 08:43:54,607][__main__][INFO] - Starting iteration 114.
[2025-11-13 08:43:54,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:54,610][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:01,605][__main__][INFO] - Number of regex retries in iteration 114: 0
[2025-11-13 08:44:01,606][__main__][INFO] - agents played in iteration 114 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:44:02,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:02,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:02,159][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:02,193][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:02,194][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:02,194][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:02,970][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:03,264][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:03,592][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:03,922][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:04,249][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:04,573][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:04,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:05,222][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:05,546][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:05,870][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:06,193][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:06,519][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:06,843][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:07,166][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:07,490][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:07,814][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:08,137][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:08,463][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:08,789][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:09,119][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:09,448][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:09,775][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:10,102][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:10,429][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:10,758][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:11,085][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:11,416][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:11,745][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:12,072][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:12,400][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:12,727][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:13,051][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:13,381][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:14,147][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:14,886][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:14,887][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:14,889][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:15,917][__main__][INFO] - Iteration 115 took 21s (32.83% Gen, 62.34% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 4m 58s. Estimated total time: 17h 45m 25s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 30s, 500 more iterations: 2h 57m 34s.
[2025-11-13 08:44:15,920][__main__][INFO] - Starting iteration 115.
[2025-11-13 08:44:15,923][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:15,923][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:22,941][__main__][INFO] - Number of regex retries in iteration 115: 0
[2025-11-13 08:44:22,942][__main__][INFO] - agents played in iteration 115 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:44:23,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,530][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:23,530][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:24,315][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:24,609][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:24,936][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:25,261][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:25,589][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:25,913][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:26,238][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:26,563][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:26,887][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:27,210][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:27,535][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:27,859][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:28,184][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:28,507][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:28,833][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:29,159][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:29,482][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:29,805][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:30,128][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:30,452][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:30,776][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:31,102][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:31,428][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:31,762][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:32,086][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:32,411][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:32,735][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:33,066][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:33,391][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:33,715][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:34,040][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:34,370][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:34,696][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:35,459][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:36,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:36,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:36,205][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:37,218][__main__][INFO] - Iteration 116 took 21s (32.96% Gen, 62.28% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 3m 58s. Estimated total time: 17h 44m 46s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 29s, 500 more iterations: 2h 57m 27s.
[2025-11-13 08:44:37,220][__main__][INFO] - Starting iteration 116.
[2025-11-13 08:44:37,224][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:37,224][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:44,237][__main__][INFO] - Number of regex retries in iteration 116: 0
[2025-11-13 08:44:44,238][__main__][INFO] - agents played in iteration 116 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:44:44,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,828][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:44,829][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:45,601][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:45,897][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:46,221][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:46,544][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:46,869][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:47,196][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:47,520][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:47,844][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:48,168][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:48,492][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:48,816][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:49,145][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:49,469][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:49,795][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:50,119][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:50,443][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:50,770][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:51,094][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:51,418][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:51,743][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:52,066][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:52,393][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:52,713][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:53,037][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:53,360][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:53,689][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:54,008][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:54,333][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:54,658][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:54,985][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:55,315][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:55,642][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:55,967][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:56,733][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:57,478][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:57,479][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:57,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:58,481][__main__][INFO] - Iteration 117 took 21s (32.99% Gen, 62.30% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 1m 45s. Estimated total time: 17h 42m 55s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 25s, 500 more iterations: 2h 57m 9s.
[2025-11-13 08:44:58,483][__main__][INFO] - Starting iteration 117.
[2025-11-13 08:44:58,487][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:58,487][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:05,536][__main__][INFO] - Number of regex retries in iteration 117: 0
[2025-11-13 08:45:05,536][__main__][INFO] - agents played in iteration 117 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:45:06,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:06,134][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:06,919][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:07,215][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:07,542][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:07,868][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:08,195][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:08,522][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:08,848][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:09,173][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:09,497][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:09,822][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:10,146][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:10,471][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:10,796][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:11,121][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:11,445][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:11,770][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:12,095][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:12,419][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:12,743][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:13,068][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:13,716][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:14,041][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:14,365][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:14,690][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:15,015][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:15,342][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:15,666][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:15,993][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:16,316][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:16,640][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:16,965][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:17,290][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:45:18,070][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:45:18,833][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:45:18,835][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:45:18,836][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:45:19,837][__main__][INFO] - Iteration 118 took 21s (33.01% Gen, 62.29% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 6m 1s. Estimated total time: 17h 47m 32s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 35s, 500 more iterations: 2h 57m 55s.
[2025-11-13 08:45:19,839][__main__][INFO] - Starting iteration 118.
[2025-11-13 08:45:19,842][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:45:19,842][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:26,448][__main__][INFO] - Number of regex retries in iteration 118: 0
[2025-11-13 08:45:26,448][__main__][INFO] - agents played in iteration 118 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:45:26,927][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:26,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,037][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:27,037][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:27,783][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:28,082][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:28,411][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:28,741][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:29,072][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:29,399][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:29,726][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:30,052][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:30,383][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:30,713][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:31,038][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:31,366][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:32,022][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:32,348][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:32,675][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:32,999][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:33,328][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:33,652][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:33,978][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:34,308][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:34,635][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:34,961][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:35,288][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:35,614][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:35,949][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:36,278][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:36,609][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:36,933][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:37,258][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:37,585][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:37,912][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:38,236][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:45:38,962][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:45:39,709][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:45:39,710][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:45:39,712][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:45:40,762][__main__][INFO] - Iteration 119 took 20s (31.58% Gen, 63.40% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 44m 11s. Estimated total time: 17h 26m 3s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 52s, 500 more iterations: 2h 54m 20s.
[2025-11-13 08:45:40,764][__main__][INFO] - Starting iteration 119.
[2025-11-13 08:45:40,768][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:45:40,768][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:47,822][__main__][INFO] - Number of regex retries in iteration 119: 0
[2025-11-13 08:45:47,822][__main__][INFO] - agents played in iteration 119 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:45:48,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,407][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:48,408][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:49,173][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:49,470][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:49,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:50,119][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:50,443][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:50,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:51,100][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:51,428][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:51,754][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:52,078][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:52,403][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:52,725][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:53,049][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:53,375][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:53,699][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:54,025][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:54,355][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:54,686][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:55,012][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:55,341][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:56,002][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:56,326][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:56,652][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:56,975][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:57,300][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:57,628][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:57,953][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:58,277][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:58,603][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:58,928][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:59,255][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:59,579][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:00,355][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:01,124][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:01,126][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:01,127][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:02,126][__main__][INFO] - Iteration 120 took 21s (33.02% Gen, 62.29% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 5m 44s. Estimated total time: 17h 47m 57s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 35s, 500 more iterations: 2h 57m 59s.
[2025-11-13 08:46:02,128][__main__][INFO] - Starting iteration 120.
[2025-11-13 08:46:02,132][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:46:02,133][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:09,314][__main__][INFO] - Number of regex retries in iteration 120: 0
[2025-11-13 08:46:09,315][__main__][INFO] - agents played in iteration 120 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:46:09,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,898][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:09,899][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:10,651][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:10,947][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:11,275][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:11,605][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:11,930][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:12,255][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:12,581][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:12,905][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:13,230][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:13,555][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:13,881][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:14,206][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:14,530][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:14,859][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:15,187][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:15,514][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:46:15,840][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:46:16,168][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:46:16,495][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:46:16,823][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:46:17,150][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:46:17,474][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:46:17,800][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:46:18,125][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:46:18,452][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:46:18,781][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:46:19,109][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:46:19,435][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:46:19,763][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:46:20,092][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:46:20,418][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:46:20,743][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:46:21,070][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:21,831][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:22,585][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:22,587][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:22,589][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:24,544][__main__][INFO] - Iteration 121 took 22s (32.04% Gen, 59.23% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 58m 2s. Estimated total time: 18h 40m 37s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 21s, 500 more iterations: 3h 6m 46s.
[2025-11-13 08:46:24,546][__main__][INFO] - Starting iteration 121.
[2025-11-13 08:46:24,549][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:46:24,550][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:31,860][__main__][INFO] - Number of regex retries in iteration 121: 0
[2025-11-13 08:46:31,861][__main__][INFO] - agents played in iteration 121 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:46:32,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,387][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,454][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:32,455][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:33,222][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:33,518][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:33,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:34,168][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:34,492][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:34,818][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:35,142][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:35,470][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:35,798][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:36,128][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:36,455][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:36,781][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:37,109][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:37,433][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:37,758][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:38,083][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:46:38,409][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:46:38,733][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:46:39,058][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:46:39,388][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:46:39,714][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:46:40,042][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:46:40,378][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:46:40,702][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:46:41,028][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:46:41,357][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:46:41,683][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:46:42,011][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:46:42,338][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:46:42,662][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:46:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:46:43,314][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:46:43,640][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:44,393][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:45,162][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:45,163][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:45,165][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:46,182][__main__][INFO] - Iteration 122 took 21s (33.80% Gen, 61.50% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 43s. Estimated total time: 18h 1m 40s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 3s, 500 more iterations: 3h 0m 16s.
[2025-11-13 08:46:46,184][__main__][INFO] - Starting iteration 122.
[2025-11-13 08:46:46,187][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:46:46,187][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:53,650][__main__][INFO] - Number of regex retries in iteration 122: 0
[2025-11-13 08:46:53,651][__main__][INFO] - agents played in iteration 122 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:46:54,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,237][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:54,238][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:54,997][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:55,292][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:55,617][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:55,945][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:56,270][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:56,595][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:56,921][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:57,250][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:57,575][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:57,901][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:58,225][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:58,552][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:58,876][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:59,202][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:59,529][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:59,855][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:00,181][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:00,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:00,837][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:01,172][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:01,496][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:01,822][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:02,148][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:02,474][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:02,798][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:03,122][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:03,447][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:03,772][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:04,098][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:04,422][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:04,746][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:05,070][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:05,395][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:06,145][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:06,915][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:06,916][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:06,918][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:07,923][__main__][INFO] - Iteration 123 took 21s (34.34% Gen, 61.04% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 23m 31s. Estimated total time: 18h 6m 50s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 13s, 500 more iterations: 3h 1m 8s.
[2025-11-13 08:47:07,925][__main__][INFO] - Starting iteration 123.
[2025-11-13 08:47:07,928][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:07,928][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:15,283][__main__][INFO] - Number of regex retries in iteration 123: 0
[2025-11-13 08:47:15,284][__main__][INFO] - agents played in iteration 123 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:47:15,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,879][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:15,879][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:16,628][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:16,924][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:47:17,249][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:47:17,576][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:47:17,901][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:47:18,230][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:47:18,554][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:47:18,879][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:47:19,203][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:47:19,527][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:47:19,850][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:47:20,175][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:47:20,499][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:47:20,823][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:47:21,148][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:47:21,472][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:21,795][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:22,119][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:22,444][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:22,770][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:23,094][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:23,417][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:23,742][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:24,066][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:24,394][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:24,717][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:25,043][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:25,367][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:25,692][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:26,019][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:26,348][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:26,672][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:26,999][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:27,793][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:28,559][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:28,560][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:28,562][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:29,540][__main__][INFO] - Iteration 124 took 21s (34.03% Gen, 61.43% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 16m 58s. Estimated total time: 18h 0m 38s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 1s, 500 more iterations: 3h 0m 6s.
[2025-11-13 08:47:29,542][__main__][INFO] - Starting iteration 124.
[2025-11-13 08:47:29,545][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:29,546][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:36,902][__main__][INFO] - Number of regex retries in iteration 124: 0
[2025-11-13 08:47:36,902][__main__][INFO] - agents played in iteration 124 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:47:37,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,499][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,499][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:37,500][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:38,252][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:38,547][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:47:38,874][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:47:39,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:47:39,524][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:47:39,849][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:47:40,174][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:47:40,499][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:47:40,823][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:47:41,147][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:47:41,472][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:47:41,797][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:47:42,122][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:47:42,445][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:47:42,770][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:47:43,095][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:43,420][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:43,746][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:44,072][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:44,396][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:44,720][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:45,045][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:45,370][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:45,695][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:46,020][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:46,344][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:46,668][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:46,993][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:47,317][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:47,643][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:47,971][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:48,306][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:48,627][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:49,395][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:50,178][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:50,180][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:50,181][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:51,192][__main__][INFO] - Iteration 125 took 21s (33.98% Gen, 61.34% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 20s. Estimated total time: 18h 2m 22s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 4s, 500 more iterations: 3h 0m 23s.
[2025-11-13 08:47:51,194][__main__][INFO] - Starting iteration 125.
[2025-11-13 08:47:51,197][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:51,197][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:58,259][__main__][INFO] - Number of regex retries in iteration 125: 0
[2025-11-13 08:47:58,260][__main__][INFO] - agents played in iteration 125 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:47:58,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,862][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:58,863][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:59,649][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:59,947][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:00,284][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:00,608][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:00,933][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:01,258][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:01,594][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:01,921][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:02,246][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:02,573][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:02,904][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:03,230][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:03,556][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:03,882][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:04,210][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:04,534][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:04,860][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:05,187][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:05,512][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:05,835][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:06,158][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:06,482][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:06,806][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:07,131][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:07,456][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:07,781][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:08,105][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:08,431][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:08,756][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:09,081][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:09,406][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:09,731][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:10,056][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:10,833][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:11,591][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:11,592][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:11,594][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:12,617][__main__][INFO] - Iteration 126 took 21s (32.97% Gen, 62.25% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 6m 39s. Estimated total time: 17h 51m 3s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 42s, 500 more iterations: 2h 58m 30s.
[2025-11-13 08:48:12,619][__main__][INFO] - Starting iteration 126.
[2025-11-13 08:48:12,622][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:12,623][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:48:20,002][__main__][INFO] - Number of regex retries in iteration 126: 0
[2025-11-13 08:48:20,003][__main__][INFO] - agents played in iteration 126 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:48:20,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,599][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:48:20,599][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:48:21,371][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:48:21,668][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:21,996][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:22,328][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:22,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:22,972][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:23,297][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:23,625][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:23,946][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:24,270][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:24,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:24,924][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:25,566][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:25,894][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:26,227][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:26,545][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:26,872][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:27,197][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:27,527][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:27,848][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:28,172][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:28,498][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:28,823][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:29,148][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:29,476][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:29,802][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:30,130][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:30,452][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:30,780][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:31,104][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:31,433][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:31,762][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:32,546][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:33,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:33,319][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:33,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:34,318][__main__][INFO] - Iteration 127 took 21s (34.01% Gen, 61.38% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 20m 4s. Estimated total time: 18h 4m 50s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 9s, 500 more iterations: 3h 0m 48s.
[2025-11-13 08:48:34,321][__main__][INFO] - Starting iteration 127.
[2025-11-13 08:48:34,323][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:34,324][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:48:41,620][__main__][INFO] - Number of regex retries in iteration 127: 0
[2025-11-13 08:48:41,620][__main__][INFO] - agents played in iteration 127 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:48:42,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:42,149][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:42,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:42,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:42,216][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:48:42,216][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:48:42,985][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:48:43,280][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:43,612][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:43,936][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:44,262][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:44,585][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:44,910][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:45,236][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:45,560][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:45,886][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:46,210][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:46,534][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:46,859][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:47,184][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:47,515][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:47,839][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:48,162][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:48,487][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:48,815][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:49,140][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:49,465][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:49,789][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:50,114][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:50,439][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:50,764][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:51,090][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:51,414][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:52,062][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:52,387][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:52,711][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:53,035][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:53,361][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:54,144][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:54,896][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:54,898][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:54,900][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:55,893][__main__][INFO] - Iteration 128 took 21s (33.82% Gen, 61.56% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 13m 23s. Estimated total time: 17h 58m 30s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 57s, 500 more iterations: 2h 59m 45s.
[2025-11-13 08:48:55,895][__main__][INFO] - Starting iteration 128.
[2025-11-13 08:48:55,898][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:55,899][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:03,082][__main__][INFO] - Number of regex retries in iteration 128: 0
[2025-11-13 08:49:03,083][__main__][INFO] - agents played in iteration 128 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:49:03,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,671][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:03,672][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:04,434][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:04,731][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:05,057][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:05,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:05,718][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:06,044][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:06,369][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:06,692][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:07,023][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:07,348][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:07,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:07,997][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:08,324][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:08,650][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:08,974][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:09,298][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:09,623][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:09,948][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:10,272][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:10,597][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:10,922][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:11,246][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:11,570][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:11,894][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:12,220][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:12,547][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:12,872][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:13,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:13,522][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:13,846][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:14,171][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:14,496][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:14,821][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:15,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:16,360][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:16,362][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:16,363][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:49:17,379][__main__][INFO] - Iteration 129 took 21s (33.44% Gen, 61.82% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 8m 38s. Estimated total time: 17h 54m 6s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 48s, 500 more iterations: 2h 59m 1s.
[2025-11-13 08:49:17,381][__main__][INFO] - Starting iteration 129.
[2025-11-13 08:49:17,384][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:49:17,385][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:24,689][__main__][INFO] - Number of regex retries in iteration 129: 0
[2025-11-13 08:49:24,690][__main__][INFO] - agents played in iteration 129 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:49:25,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,307][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:25,307][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:26,077][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:26,374][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:26,701][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:27,026][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:27,350][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:27,676][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:28,006][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:28,332][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:28,658][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:28,985][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:29,309][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:29,633][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:29,957][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:30,283][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:30,610][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:30,934][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:31,258][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:31,583][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:31,908][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:32,232][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:32,557][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:32,883][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:33,209][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:33,534][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:33,860][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:34,187][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:34,511][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:34,835][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:35,159][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:35,486][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:35,811][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:36,137][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:36,461][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:37,204][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:37,949][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:37,951][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:37,953][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:49:38,942][__main__][INFO] - Iteration 130 took 21s (33.88% Gen, 61.52% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 12m 4s. Estimated total time: 17h 57m 55s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 55s, 500 more iterations: 2h 59m 39s.
[2025-11-13 08:49:38,944][__main__][INFO] - Starting iteration 130.
[2025-11-13 08:49:38,949][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:49:38,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:46,236][__main__][INFO] - Number of regex retries in iteration 130: 0
[2025-11-13 08:49:46,237][__main__][INFO] - agents played in iteration 130 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:49:46,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,835][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:46,836][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:47,605][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:47,901][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:48,229][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:48,555][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:48,882][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:49,217][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:49,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:49,870][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:50,196][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:50,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:50,846][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:51,171][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:51,496][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:51,822][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:52,148][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:52,473][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:52,798][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:53,126][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:53,450][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:53,778][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:54,104][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:54,431][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:54,757][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:55,082][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:55,407][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:55,732][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:56,056][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:56,384][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:56,710][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:57,036][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:57,361][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:57,689][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:58,015][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:58,786][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:59,544][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:59,546][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:59,548][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:01,485][__main__][INFO] - Iteration 131 took 22s (32.34% Gen, 59.06% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 0m 38s. Estimated total time: 18h 46m 50s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 48s.
[2025-11-13 08:50:01,488][__main__][INFO] - Starting iteration 131.
[2025-11-13 08:50:01,492][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:01,492][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:08,845][__main__][INFO] - Number of regex retries in iteration 131: 0
[2025-11-13 08:50:08,846][__main__][INFO] - agents played in iteration 131 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:50:09,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,813][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:09,813][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:10,570][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:10,865][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:11,191][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:11,516][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:11,840][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:12,164][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:12,489][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:12,815][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:13,140][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:13,464][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:13,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:14,114][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:14,438][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:14,762][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:15,087][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:15,415][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:15,742][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:16,067][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:50:16,391][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:50:16,719][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:50:17,044][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:50:17,370][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:50:17,695][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:50:18,024][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:50:18,351][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:50:18,678][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:50:19,003][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:50:19,328][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:50:19,652][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:50:19,977][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:50:20,301][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:50:20,625][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:50:20,954][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:50:21,701][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:50:22,447][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:50:22,449][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:50:22,451][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:23,424][__main__][INFO] - Iteration 132 took 21s (33.53% Gen, 62.03% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 30m 4s. Estimated total time: 18h 16m 38s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 33s, 500 more iterations: 3h 2m 46s.
[2025-11-13 08:50:23,426][__main__][INFO] - Starting iteration 132.
[2025-11-13 08:50:23,428][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:23,429][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:30,606][__main__][INFO] - Number of regex retries in iteration 132: 0
[2025-11-13 08:50:30,607][__main__][INFO] - agents played in iteration 132 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:50:31,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:31,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:31,967][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:32,265][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:32,599][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:32,926][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:33,257][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:33,588][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:33,923][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:34,248][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:34,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:34,897][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:35,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:35,559][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:35,886][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:36,221][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:36,548][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:36,874][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:37,204][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:37,530][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:50:37,860][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:50:38,188][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:50:38,512][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:50:38,839][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:50:39,170][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:50:39,502][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:50:39,826][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:50:40,155][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:50:40,482][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:50:40,807][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:50:41,137][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:50:41,462][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:50:41,789][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:50:42,114][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:50:42,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:50:43,189][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:50:43,944][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:50:43,945][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:50:43,947][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:44,989][__main__][INFO] - Iteration 133 took 21s (33.29% Gen, 61.87% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 11m 7s. Estimated total time: 17h 58m 3s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 56s, 500 more iterations: 2h 59m 40s.
[2025-11-13 08:50:44,991][__main__][INFO] - Starting iteration 133.
[2025-11-13 08:50:44,994][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:44,995][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:52,819][__main__][INFO] - Number of regex retries in iteration 133: 0
[2025-11-13 08:50:52,820][__main__][INFO] - agents played in iteration 133 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:50:53,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,420][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:53,421][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:54,170][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:54,466][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:54,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:55,119][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:55,444][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:55,769][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:56,094][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:56,421][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:56,747][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:57,073][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:57,399][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:57,728][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:58,051][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:58,376][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:58,702][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:59,026][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:59,351][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:59,679][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:51:00,013][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:00,337][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:00,662][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:00,985][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:01,309][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:01,636][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:01,959][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:02,283][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:02,608][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:02,931][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:03,255][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:03,578][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:03,904][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:04,228][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:04,552][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:05,309][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:06,056][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:06,057][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:06,059][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:07,041][__main__][INFO] - Iteration 134 took 22s (35.49% Gen, 60.05% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 35m 4s. Estimated total time: 18h 22m 22s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 44s, 500 more iterations: 3h 3m 43s.
[2025-11-13 08:51:07,043][__main__][INFO] - Starting iteration 134.
[2025-11-13 08:51:07,046][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:07,047][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:14,546][__main__][INFO] - Number of regex retries in iteration 134: 0
[2025-11-13 08:51:14,546][__main__][INFO] - agents played in iteration 134 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:51:15,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,140][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:15,140][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:15,916][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:51:16,212][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:51:16,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:51:16,864][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:51:17,191][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:51:17,515][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:51:17,841][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:51:18,166][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:51:18,491][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:51:18,816][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:51:19,141][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:51:19,466][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:51:19,790][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:51:20,114][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:51:20,440][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:51:20,764][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:51:21,088][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:51:21,416][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:51:21,741][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:22,065][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:22,391][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:22,717][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:23,043][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:23,368][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:23,694][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:24,020][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:24,345][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:24,671][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:24,996][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:25,322][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:25,646][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:25,970][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:26,300][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:27,064][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:27,825][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:27,827][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:27,829][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:28,824][__main__][INFO] - Iteration 135 took 21s (34.44% Gen, 60.99% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 21m 15s. Estimated total time: 18h 8m 55s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 29s.
[2025-11-13 08:51:28,826][__main__][INFO] - Starting iteration 135.
[2025-11-13 08:51:28,830][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:28,830][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:36,462][__main__][INFO] - Number of regex retries in iteration 135: 0
[2025-11-13 08:51:36,462][__main__][INFO] - agents played in iteration 135 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:51:36,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:36,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,063][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:37,063][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:37,832][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:51:38,130][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:51:38,456][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:51:38,786][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:51:39,116][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:51:39,444][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:51:39,769][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:51:40,092][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:51:40,416][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:51:40,740][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:51:41,063][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:51:41,387][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:51:41,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:51:42,034][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:51:42,361][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:51:42,685][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:51:43,008][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:51:43,332][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:51:43,660][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:43,987][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:44,313][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:44,639][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:44,967][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:45,294][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:45,620][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:45,946][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:46,275][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:46,600][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:46,923][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:47,248][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:47,573][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:47,901][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:48,225][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:48,982][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:49,738][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:49,739][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:49,741][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:50,731][__main__][INFO] - Iteration 136 took 21s (34.84% Gen, 60.63% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 27m 4s. Estimated total time: 18h 15m 5s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 30s, 500 more iterations: 3h 2m 30s.
[2025-11-13 08:51:50,733][__main__][INFO] - Starting iteration 136.
[2025-11-13 08:51:50,736][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:50,736][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:58,375][__main__][INFO] - Number of regex retries in iteration 136: 0
[2025-11-13 08:51:58,376][__main__][INFO] - agents played in iteration 136 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:51:58,855][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,889][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,956][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:58,957][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:59,720][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:00,015][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:00,347][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:00,665][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:00,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:01,316][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:01,647][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:01,965][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:02,291][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:02,614][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:02,942][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:03,268][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:03,592][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:03,920][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:04,245][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:04,570][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:04,895][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:05,221][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:05,547][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:05,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:06,196][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:06,519][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:06,844][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:07,170][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:07,494][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:07,818][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:08,141][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:08,466][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:08,790][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:09,115][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:09,439][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:09,762][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:10,087][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:10,846][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:11,613][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:11,615][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:11,616][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:12,629][__main__][INFO] - Iteration 137 took 21s (34.90% Gen, 60.48% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 26m 17s. Estimated total time: 18h 14m 41s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 29s, 500 more iterations: 3h 2m 26s.
[2025-11-13 08:52:12,631][__main__][INFO] - Starting iteration 137.
[2025-11-13 08:52:12,635][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:12,635][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:52:20,458][__main__][INFO] - Number of regex retries in iteration 137: 0
[2025-11-13 08:52:20,458][__main__][INFO] - agents played in iteration 137 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:52:20,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:20,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:21,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:21,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:21,045][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:52:21,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:52:21,816][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:22,111][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:22,437][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:22,763][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:23,087][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:23,412][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:23,737][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:24,061][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:24,386][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:24,711][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:25,036][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:25,361][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:25,685][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:26,013][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:26,339][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:26,663][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:26,989][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:27,314][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:27,639][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:27,965][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:28,288][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:28,613][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:28,939][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:29,263][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:29,586][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:29,910][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:30,235][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:30,560][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:30,885][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:31,210][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:31,534][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:31,859][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:32,182][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:32,948][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:33,718][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:33,720][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:33,721][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:34,720][__main__][INFO] - Iteration 138 took 22s (35.42% Gen, 60.05% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 35m 32s. Estimated total time: 18h 24m 17s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 48s, 500 more iterations: 3h 4m 2s.
[2025-11-13 08:52:34,722][__main__][INFO] - Starting iteration 138.
[2025-11-13 08:52:34,724][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:34,725][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:52:42,489][__main__][INFO] - Number of regex retries in iteration 138: 0
[2025-11-13 08:52:42,490][__main__][INFO] - agents played in iteration 138 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:52:43,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:43,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:43,075][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:43,108][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:43,109][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:52:43,109][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:52:43,850][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:44,148][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:44,473][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:44,799][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:45,124][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:45,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:45,777][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:46,102][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:46,426][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:46,754][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:47,088][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:47,415][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:47,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:48,065][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:48,392][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:48,717][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:49,042][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:49,369][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:49,693][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:50,340][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:50,669][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:50,990][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:51,315][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:51,639][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:51,963][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:52,287][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:52,611][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:52,936][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:53,262][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:53,585][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:53,910][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:54,236][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:55,001][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:55,753][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:55,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:55,760][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:56,719][__main__][INFO] - Iteration 139 took 21s (35.30% Gen, 60.33% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 30m 36s. Estimated total time: 18h 19m 44s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 39s, 500 more iterations: 3h 3m 17s.
[2025-11-13 08:52:56,721][__main__][INFO] - Starting iteration 139.
[2025-11-13 08:52:56,724][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:56,724][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:04,251][__main__][INFO] - Number of regex retries in iteration 139: 0
[2025-11-13 08:53:04,252][__main__][INFO] - agents played in iteration 139 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:53:04,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,819][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:04,819][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:05,572][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:05,868][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:06,194][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:06,519][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:06,846][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:07,170][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:07,494][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:07,820][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:08,145][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:08,469][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:08,795][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:09,119][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:09,443][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:09,767][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:10,091][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:10,415][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:10,738][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:11,062][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:11,386][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:11,712][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:12,036][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:12,359][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:12,686][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:13,014][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:13,340][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:13,667][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:13,991][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:14,314][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:53:14,639][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:53:14,963][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:53:15,286][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:53:15,611][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:53:15,943][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:53:16,688][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:53:17,450][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:53:17,451][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:53:17,453][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:53:18,462][__main__][INFO] - Iteration 140 took 21s (34.63% Gen, 60.72% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 17m 27s. Estimated total time: 18h 6m 57s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 13s, 500 more iterations: 3h 1m 9s.
[2025-11-13 08:53:18,464][__main__][INFO] - Starting iteration 140.
[2025-11-13 08:53:18,466][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:53:18,467][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:26,216][__main__][INFO] - Number of regex retries in iteration 140: 0
[2025-11-13 08:53:26,216][__main__][INFO] - agents played in iteration 140 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:53:26,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,808][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,809][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:26,809][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:27,569][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:27,863][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:28,189][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:28,513][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:28,841][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:29,165][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:29,492][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:29,818][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:30,147][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:30,476][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:30,807][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:31,132][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:31,457][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:31,782][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:32,107][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:32,438][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:32,763][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:33,090][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:33,416][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:33,742][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:34,070][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:34,725][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:35,051][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:35,375][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:35,702][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:36,026][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:36,350][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:53:36,674][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:53:36,997][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:53:37,322][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:53:37,648][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:53:37,979][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:53:38,740][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:53:39,509][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:53:39,510][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:53:39,512][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:53:41,481][__main__][INFO] - Iteration 141 took 23s (33.67% Gen, 57.77% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 20m 52s. Estimated total time: 19h 10m 45s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 47s.
[2025-11-13 08:53:41,483][__main__][INFO] - Starting iteration 141.
[2025-11-13 08:53:41,486][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:53:41,486][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:49,744][__main__][INFO] - Number of regex retries in iteration 141: 0
[2025-11-13 08:53:49,744][__main__][INFO] - agents played in iteration 141 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:53:50,230][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,331][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:50,331][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:51,080][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:51,379][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:51,703][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:52,028][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:52,352][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:52,678][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:53,003][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:53,327][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:53,651][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:53,975][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:54,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:54,621][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:54,948][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:55,271][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:55,597][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:55,921][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:56,245][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:56,569][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:56,892][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:57,217][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:57,542][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:57,864][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:58,190][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:58,514][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:58,838][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:59,162][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:59,486][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:59,811][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:00,135][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:00,459][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:00,789][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:01,113][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:01,438][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:02,192][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:02,966][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:02,967][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:02,969][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:03,972][__main__][INFO] - Iteration 142 took 22s (36.72% Gen, 58.81% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 54m 7s. Estimated total time: 18h 44m 22s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 28s, 500 more iterations: 3h 7m 23s.
[2025-11-13 08:54:03,975][__main__][INFO] - Starting iteration 142.
[2025-11-13 08:54:03,979][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:03,980][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:11,490][__main__][INFO] - Number of regex retries in iteration 142: 0
[2025-11-13 08:54:11,491][__main__][INFO] - agents played in iteration 142 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:54:11,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,098][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:12,098][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:12,870][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:13,168][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:13,494][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:13,819][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:14,143][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:14,469][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:14,793][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:15,117][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:15,441][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:54:15,766][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:54:16,090][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:54:16,416][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:54:16,741][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:54:17,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:54:17,397][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:54:17,721][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:54:18,046][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:54:18,371][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:54:18,695][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:54:19,020][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:54:19,344][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:54:19,671][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:54:19,996][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:54:20,320][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:54:20,645][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:54:20,971][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:54:21,295][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:54:21,622][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:21,950][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:22,275][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:22,599][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:22,924][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:23,249][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:24,007][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:24,769][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:24,770][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:24,772][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:25,763][__main__][INFO] - Iteration 143 took 21s (34.48% Gen, 60.96% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 37s. Estimated total time: 18h 9m 14s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 18s, 500 more iterations: 3h 1m 32s.
[2025-11-13 08:54:25,765][__main__][INFO] - Starting iteration 143.
[2025-11-13 08:54:25,769][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:25,770][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:33,572][__main__][INFO] - Number of regex retries in iteration 143: 0
[2025-11-13 08:54:33,573][__main__][INFO] - agents played in iteration 143 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:54:34,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,174][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:34,175][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:34,948][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:35,245][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:35,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:35,900][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:36,230][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:36,557][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:36,886][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:37,216][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:37,543][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:54:37,871][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:54:38,201][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:54:38,528][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:54:38,855][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:54:39,184][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:54:39,511][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:54:39,836][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:54:40,161][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:54:40,484][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:54:40,809][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:54:41,134][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:54:41,458][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:54:41,783][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:54:42,107][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:54:42,432][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:54:42,756][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:54:43,081][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:54:43,407][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:54:43,732][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:44,057][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:44,380][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:44,706][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:45,032][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:45,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:46,111][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:46,868][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:46,869][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:46,871][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:47,876][__main__][INFO] - Iteration 144 took 22s (35.29% Gen, 60.15% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 34m 25s. Estimated total time: 18h 25m 24s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 50s, 500 more iterations: 3h 4m 14s.
[2025-11-13 08:54:47,878][__main__][INFO] - Starting iteration 144.
[2025-11-13 08:54:47,881][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:47,882][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:55,956][__main__][INFO] - Number of regex retries in iteration 144: 0
[2025-11-13 08:54:55,957][__main__][INFO] - agents played in iteration 144 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:54:56,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,556][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:56,556][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:57,320][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:57,615][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:57,940][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:58,267][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:58,591][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:58,918][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:59,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:59,911][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:00,243][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:00,573][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:00,897][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:01,224][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:01,551][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:01,882][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:02,208][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:02,533][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:02,863][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:03,195][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:03,523][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:03,850][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:04,178][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:04,508][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:04,838][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:05,166][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:05,496][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:05,828][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:06,158][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:06,485][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:06,812][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:07,139][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:07,463][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:07,793][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:08,550][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:09,309][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:09,311][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:09,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:10,309][__main__][INFO] - Iteration 145 took 22s (36.00% Gen, 59.54% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 50m 5s. Estimated total time: 18h 41m 26s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 22s, 500 more iterations: 3h 6m 54s.
[2025-11-13 08:55:10,312][__main__][INFO] - Starting iteration 145.
[2025-11-13 08:55:10,315][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:10,316][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:55:18,113][__main__][INFO] - Number of regex retries in iteration 145: 0
[2025-11-13 08:55:18,114][__main__][INFO] - agents played in iteration 145 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:55:18,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,711][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,712][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:55:18,712][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:55:19,481][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:55:19,779][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:55:20,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:55:20,426][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:55:20,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:55:21,076][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:55:21,403][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:55:21,728][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:55:22,052][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:22,377][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:22,701][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:23,026][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:23,350][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:23,674][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:23,999][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:24,323][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:24,647][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:24,972][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:25,296][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:25,621][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:25,947][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:26,272][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:26,598][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:26,923][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:27,248][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:27,575][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:27,900][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:28,224][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:28,548][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:28,875][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:29,199][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:29,524][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:29,852][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:30,609][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:31,361][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:31,362][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:31,364][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:32,364][__main__][INFO] - Iteration 146 took 22s (35.37% Gen, 60.09% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 30m 44s. Estimated total time: 18h 22m 28s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 44s, 500 more iterations: 3h 3m 44s.
[2025-11-13 08:55:32,366][__main__][INFO] - Starting iteration 146.
[2025-11-13 08:55:32,371][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:32,371][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:55:40,202][__main__][INFO] - Number of regex retries in iteration 146: 0
[2025-11-13 08:55:40,202][__main__][INFO] - agents played in iteration 146 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:55:40,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,786][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:55:40,787][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:55:41,547][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:55:41,842][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:55:42,168][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:55:42,496][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:55:42,821][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:55:43,146][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:55:43,473][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:55:43,797][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:55:44,121][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:44,446][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:44,773][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:45,100][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:45,425][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:45,749][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:46,073][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:46,398][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:46,725][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:47,050][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:47,385][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:47,702][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:48,028][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:48,353][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:48,677][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:49,002][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:49,651][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:49,980][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:50,303][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:50,627][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:50,951][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:51,280][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:51,601][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:51,926][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:52,690][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:53,445][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:53,447][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:53,448][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:54,406][__main__][INFO] - Iteration 147 took 22s (35.54% Gen, 60.11% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 44s. Estimated total time: 18h 21m 49s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 43s, 500 more iterations: 3h 3m 38s.
[2025-11-13 08:55:54,409][__main__][INFO] - Starting iteration 147.
[2025-11-13 08:55:54,412][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:54,412][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:02,459][__main__][INFO] - Number of regex retries in iteration 147: 0
[2025-11-13 08:56:02,459][__main__][INFO] - agents played in iteration 147 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:56:02,948][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:02,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:03,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:03,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:03,050][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:03,051][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:03,824][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:04,119][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:04,443][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:04,775][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:05,098][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:05,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:05,750][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:06,084][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:06,410][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:06,735][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:07,062][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:07,391][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:07,715][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:08,040][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:08,371][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:08,701][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:09,029][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:09,356][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:09,681][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:10,006][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:10,331][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:10,657][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:10,981][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:11,305][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:11,630][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:11,955][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:12,280][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:12,610][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:12,935][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:13,262][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:13,587][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:13,912][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:14,242][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:15,024][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:15,772][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:15,776][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:15,779][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:16,716][__main__][INFO] - Iteration 148 took 22s (36.08% Gen, 59.72% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 42m 47s. Estimated total time: 18h 35m 15s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 10s, 500 more iterations: 3h 5m 52s.
[2025-11-13 08:56:16,718][__main__][INFO] - Starting iteration 148.
[2025-11-13 08:56:16,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:56:16,721][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:24,561][__main__][INFO] - Number of regex retries in iteration 148: 0
[2025-11-13 08:56:24,562][__main__][INFO] - agents played in iteration 148 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:56:25,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:25,075][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:25,108][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:25,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:25,142][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:25,143][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:26,201][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:26,526][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:26,850][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:27,175][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:27,499][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:27,824][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:28,149][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:28,475][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:28,802][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:29,126][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:29,451][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:29,775][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:30,100][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:30,424][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:30,748][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:31,075][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:31,404][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:31,732][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:32,057][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:32,383][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:32,710][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:33,033][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:33,358][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:33,684][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:34,009][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:34,334][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:34,659][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:35,315][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:35,646][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:35,976][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:36,302][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:37,047][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:37,774][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:37,775][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:37,776][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:38,711][__main__][INFO] - Iteration 149 took 21s (35.65% Gen, 60.09% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 26m 44s. Estimated total time: 18h 19m 34s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 39s, 500 more iterations: 3h 3m 15s.
[2025-11-13 08:56:38,713][__main__][INFO] - Starting iteration 149.
[2025-11-13 08:56:38,716][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:56:38,717][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:46,584][__main__][INFO] - Number of regex retries in iteration 149: 0
[2025-11-13 08:56:46,584][__main__][INFO] - agents played in iteration 149 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:56:47,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:47,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:47,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:47,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:47,165][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:47,165][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:47,926][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:48,222][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:48,546][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:48,872][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:49,197][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:49,522][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:49,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:50,171][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:50,496][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:50,822][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:51,151][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:51,475][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:51,805][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:52,130][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:52,457][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:52,782][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:53,114][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:53,433][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:53,757][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:54,086][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:54,411][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:54,738][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:55,062][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:55,387][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:55,713][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:56,042][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:56,366][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:56,689][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:57,022][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:57,342][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:57,667][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:57,996][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:58,328][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:59,057][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:59,815][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:59,817][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:59,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:00,754][__main__][INFO] - Iteration 150 took 22s (35.70% Gen, 60.06% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 28m 43s. Estimated total time: 18h 21m 55s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 43s, 500 more iterations: 3h 3m 39s.
[2025-11-13 08:57:00,756][__main__][INFO] - Starting iteration 150.
[2025-11-13 08:57:00,759][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:57:00,759][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:08,724][__main__][INFO] - Number of regex retries in iteration 150: 0
[2025-11-13 08:57:08,725][__main__][INFO] - agents played in iteration 150 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:57:09,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:09,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:09,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:09,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:09,300][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:09,300][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:10,044][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:10,341][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:10,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:10,990][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:11,316][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:11,645][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:11,972][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:12,296][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:12,621][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:12,947][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:13,273][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:13,600][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:13,925][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:57:14,250][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:57:14,577][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:57:14,903][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:57:15,229][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:57:15,554][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:57:15,878][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:57:16,202][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:57:16,531][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:57:16,855][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:57:17,179][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:57:17,503][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:57:17,828][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:57:18,152][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:57:18,478][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:57:18,805][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:57:19,131][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:57:19,458][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:57:19,783][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:57:20,109][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:57:20,433][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:57:21,202][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:57:21,970][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:57:21,971][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:57:21,973][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:23,820][__main__][INFO] - Iteration 151 took 23s (34.54% Gen, 57.45% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 19m 29s. Estimated total time: 19h 13m 4s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 26s, 500 more iterations: 3h 12m 10s.
[2025-11-13 08:57:23,822][__main__][INFO] - Starting iteration 151.
[2025-11-13 08:57:23,825][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:57:23,825][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:29,916][mllm.models.large_language_model_local][WARNING] - Response <B>
user
Last round, the other agent played <A>. did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 08:57:32,548][__main__][INFO] - Number of regex retries in iteration 151: 1
[2025-11-13 08:57:32,549][__main__][INFO] - agents played in iteration 151 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:57:33,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:33,074][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:33,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:33,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:33,141][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:33,142][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:33,858][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:34,154][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:34,480][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:34,809][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:35,130][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:35,458][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:35,786][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:36,125][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:36,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:36,772][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:37,096][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:37,428][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:37,746][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:57:38,076][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:57:38,401][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:57:38,734][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:57:39,057][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:57:39,383][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:57:39,708][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:57:40,037][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:57:40,357][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:57:40,685][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:57:41,010][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:57:41,338][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:57:41,662][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:57:41,987][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:57:42,313][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:57:42,640][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:57:42,968][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:57:43,297][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:57:43,624][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:57:43,950][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:57:44,283][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:57:45,033][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:57:45,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:57:45,771][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:57:45,772][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:46,704][__main__][INFO] - Iteration 152 took 22s (38.13% Gen, 57.80% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 10m 1s. Estimated total time: 19h 3m 59s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 7s, 500 more iterations: 3h 10m 39s.
[2025-11-13 08:57:46,708][__main__][INFO] - Starting iteration 152.
[2025-11-13 08:57:46,712][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:57:46,713][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:54,549][__main__][INFO] - Number of regex retries in iteration 152: 0
[2025-11-13 08:57:54,549][__main__][INFO] - agents played in iteration 152 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:57:55,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:55,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:55,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:55,201][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:55,201][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:55,202][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:55,930][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:56,226][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:56,551][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:56,876][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:57,202][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:57,526][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:57,852][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:58,176][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:58,504][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:58,827][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:59,154][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:59,479][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:59,811][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:00,132][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:00,781][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:01,110][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:01,431][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:58:01,758][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:58:02,085][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:58:02,418][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:58:02,739][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:58:03,066][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:58:03,398][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:58:03,730][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:58:04,052][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:58:04,377][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:58:04,703][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:05,028][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:05,353][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:05,679][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:06,005][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:06,333][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:07,099][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:07,843][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:07,845][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:07,846][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:08,784][__main__][INFO] - Iteration 153 took 22s (35.50% Gen, 60.24% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 19s. Estimated total time: 18h 23m 39s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 47s, 500 more iterations: 3h 3m 56s.
[2025-11-13 08:58:08,786][__main__][INFO] - Starting iteration 153.
[2025-11-13 08:58:08,788][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:08,789][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:16,495][__main__][INFO] - Number of regex retries in iteration 153: 0
[2025-11-13 08:58:16,496][__main__][INFO] - agents played in iteration 153 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:58:16,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,095][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:17,096][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:17,096][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:17,829][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:58:18,124][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:58:18,449][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:58:18,774][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:58:19,101][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:58:19,426][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:58:19,754][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:58:20,081][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:58:20,411][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:58:20,738][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:58:21,065][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:58:21,390][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:58:21,721][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:22,048][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:22,384][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:22,711][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:23,037][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:23,361][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:58:23,687][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:58:24,017][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:58:24,346][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:58:24,672][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:58:25,001][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:58:25,329][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:58:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:58:25,979][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:58:26,306][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:58:26,633][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:26,961][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:27,287][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:27,614][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:27,940][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:28,267][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:29,045][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:29,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:29,769][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:29,770][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:30,724][__main__][INFO] - Iteration 154 took 21s (35.13% Gen, 60.51% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 22m 6s. Estimated total time: 18h 16m 48s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 33s, 500 more iterations: 3h 2m 48s.
[2025-11-13 08:58:30,726][__main__][INFO] - Starting iteration 154.
[2025-11-13 08:58:30,728][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:30,729][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:38,451][__main__][INFO] - Number of regex retries in iteration 154: 0
[2025-11-13 08:58:38,452][__main__][INFO] - agents played in iteration 154 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:58:38,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:38,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:39,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:39,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:39,045][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:39,045][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:39,790][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:58:40,084][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:58:40,413][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:58:40,740][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:58:41,065][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:58:41,389][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:58:41,714][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:58:42,043][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:58:42,367][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:58:42,695][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:58:43,021][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:58:43,347][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:58:43,673][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:43,997][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:44,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:44,651][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:44,976][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:45,305][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:58:45,630][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:58:45,960][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:58:46,287][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:58:46,614][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:58:46,940][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:58:47,263][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:58:47,589][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:58:47,914][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:58:48,241][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:58:48,566][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:48,891][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:49,217][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:49,543][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:49,870][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:50,196][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:50,971][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:51,719][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:51,721][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:51,722][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:52,689][__main__][INFO] - Iteration 155 took 21s (35.17% Gen, 60.42% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 23m 0s. Estimated total time: 18h 18m 3s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 36s, 500 more iterations: 3h 3m 0s.
[2025-11-13 08:58:52,691][__main__][INFO] - Starting iteration 155.
[2025-11-13 08:58:52,693][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:52,694][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:00,266][__main__][INFO] - Number of regex retries in iteration 155: 0
[2025-11-13 08:59:00,267][__main__][INFO] - agents played in iteration 155 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:59:00,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:00,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:00,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:00,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:00,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:00,858][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:01,903][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:02,228][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:02,557][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:02,882][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:03,208][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:03,535][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:03,863][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:04,190][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:04,522][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:04,853][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:59:05,183][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:59:05,509][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:59:05,837][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:59:06,165][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:59:06,496][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:59:06,823][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:59:07,149][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:07,475][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:07,800][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:08,127][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:08,453][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:08,779][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:09,104][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:09,430][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:09,755][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:10,081][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:10,407][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:10,737][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:11,059][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:11,387][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:11,714][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:12,041][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:12,801][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:13,564][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:13,565][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:13,567][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:14,549][__main__][INFO] - Iteration 156 took 21s (34.65% Gen, 60.85% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 17m 24s. Estimated total time: 18h 12m 50s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 25s, 500 more iterations: 3h 2m 8s.
[2025-11-13 08:59:14,551][__main__][INFO] - Starting iteration 156.
[2025-11-13 08:59:14,554][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:14,554][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:22,384][__main__][INFO] - Number of regex retries in iteration 156: 0
[2025-11-13 08:59:22,384][__main__][INFO] - agents played in iteration 156 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:59:22,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:22,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:23,016][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:23,017][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:23,017][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:23,741][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:24,035][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:24,359][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:24,683][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:25,007][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:25,332][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:25,658][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:25,982][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:26,307][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:26,632][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:26,957][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:59:27,287][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:59:27,614][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:59:27,939][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:59:28,265][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:59:28,591][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:59:28,916][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:59:29,245][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:29,569][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:29,896][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:30,221][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:30,547][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:30,874][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:31,199][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:31,524][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:31,852][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:32,174][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:32,500][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:32,824][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:33,154][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:33,478][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:33,810][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:34,136][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:34,925][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:35,649][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:35,651][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:35,653][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:36,584][__main__][INFO] - Iteration 157 took 22s (35.54% Gen, 60.23% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 25m 44s. Estimated total time: 18h 21m 32s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 43s, 500 more iterations: 3h 3m 35s.
[2025-11-13 08:59:36,586][__main__][INFO] - Starting iteration 157.
[2025-11-13 08:59:36,589][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:36,589][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:43,850][__main__][INFO] - Number of regex retries in iteration 157: 0
[2025-11-13 08:59:43,850][__main__][INFO] - agents played in iteration 157 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 08:59:44,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:44,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:44,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:44,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:44,427][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:44,428][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:45,193][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:45,489][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:45,814][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:46,146][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:46,469][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:46,797][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:47,128][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:47,460][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:47,791][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:48,123][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:48,447][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:59:48,773][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:59:49,110][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:59:49,441][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:59:49,765][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:59:50,091][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:59:50,423][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:59:50,748][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:51,074][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:51,400][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:51,729][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:52,055][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:52,381][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:52,707][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:53,032][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:53,357][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:53,683][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:54,009][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:54,335][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:54,661][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:54,986][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:55,312][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:55,637][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:56,364][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:57,114][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:57,116][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:57,117][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:58,115][__main__][INFO] - Iteration 158 took 21s (33.73% Gen, 61.63% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 0m 12s. Estimated total time: 17h 56m 21s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 52s, 500 more iterations: 2h 59m 23s.
[2025-11-13 08:59:58,117][__main__][INFO] - Starting iteration 158.
[2025-11-13 08:59:58,121][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:58,121][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:05,942][__main__][INFO] - Number of regex retries in iteration 158: 0
[2025-11-13 09:00:05,943][__main__][INFO] - agents played in iteration 158 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:00:06,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:06,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:06,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:06,535][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:06,536][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:06,536][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:07,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:00:07,615][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:00:07,943][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:00:08,270][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:00:08,603][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:00:08,934][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:00:09,261][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:00:09,588][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:00:09,916][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:00:10,245][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:00:10,570][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:10,895][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:11,229][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:11,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:11,873][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:12,198][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:12,529][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:12,847][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:13,173][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:13,501][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:13,840][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:14,158][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:14,489][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:14,815][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:15,140][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:15,471][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:15,799][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:00:16,127][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:00:16,454][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:00:16,782][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:00:17,108][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:00:17,433][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:00:17,758][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:00:18,514][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:00:19,237][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:00:19,239][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:00:19,241][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:00:20,231][__main__][INFO] - Iteration 159 took 22s (35.37% Gen, 60.14% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 3s. Estimated total time: 18h 25m 34s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 51s, 500 more iterations: 3h 4m 15s.
[2025-11-13 09:00:20,234][__main__][INFO] - Starting iteration 159.
[2025-11-13 09:00:20,237][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 09:00:20,238][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:27,776][__main__][INFO] - Number of regex retries in iteration 159: 0
[2025-11-13 09:00:27,776][__main__][INFO] - agents played in iteration 159 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:00:28,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:28,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:28,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:28,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:28,381][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:28,381][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:29,158][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:00:29,455][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:00:29,781][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:00:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:00:30,432][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:00:30,758][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:00:31,083][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:00:31,409][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:00:31,735][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:00:32,059][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:00:32,385][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:32,713][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:33,039][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:33,365][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:33,689][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:34,015][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:34,340][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:34,664][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:34,989][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:35,314][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:35,639][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:35,964][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:36,289][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:36,614][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:36,941][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:37,265][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:37,591][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:00:37,916][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:00:38,241][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:00:38,567][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:00:38,892][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:00:39,217][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:00:39,540][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:00:40,297][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:00:41,039][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:00:41,041][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:00:41,042][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:00:42,046][__main__][INFO] - Iteration 160 took 21s (34.56% Gen, 60.83% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 13m 35s. Estimated total time: 18h 10m 28s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 20s, 500 more iterations: 3h 1m 44s.
[2025-11-13 09:00:42,048][__main__][INFO] - Starting iteration 160.
[2025-11-13 09:00:42,051][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 09:00:42,051][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:49,901][__main__][INFO] - Number of regex retries in iteration 160: 0
[2025-11-13 09:00:49,901][__main__][INFO] - agents played in iteration 160 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:00:50,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:50,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:50,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:50,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:50,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:50,496][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:51,273][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:00:51,570][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:00:51,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:00:52,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:00:52,557][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:00:52,884][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:00:53,210][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:00:53,539][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:00:53,866][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:00:54,192][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:00:54,519][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:54,852][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:55,178][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:55,503][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:55,835][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:56,154][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:56,479][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:56,810][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:57,142][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:57,461][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:57,785][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:58,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:58,442][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:58,760][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:59,085][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:59,410][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:59,742][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:00,060][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:00,386][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:00,712][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:01,037][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:01,364][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:01,689][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:02,429][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:03,184][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:03,186][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:03,188][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:05,174][__main__][INFO] - Iteration 161 took 23s (33.95% Gen, 57.46% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 18m 56s. Estimated total time: 19h 16m 12s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 32s, 500 more iterations: 3h 12m 42s.
[2025-11-13 09:01:05,177][__main__][INFO] - Starting iteration 161.
[2025-11-13 09:01:05,180][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:05,180][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:13,309][__main__][INFO] - Number of regex retries in iteration 161: 0
[2025-11-13 09:01:13,310][__main__][INFO] - agents played in iteration 161 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:01:13,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:13,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:13,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:13,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:13,895][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:13,895][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:14,656][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:14,953][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:15,279][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:15,605][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:15,931][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:01:16,257][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:01:16,582][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:01:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:01:17,236][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:01:17,561][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:01:17,886][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:01:18,215][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:01:18,552][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:01:18,872][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:01:19,198][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:01:19,525][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:01:19,859][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:01:20,180][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:01:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:01:20,833][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:01:21,168][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:01:21,498][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:01:21,828][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:01:22,154][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:01:22,479][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:01:22,809][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:01:23,133][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:23,458][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:23,782][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:24,113][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:24,439][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:24,766][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:25,092][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:25,814][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:26,550][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:26,552][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:26,553][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:27,502][__main__][INFO] - Iteration 162 took 22s (36.42% Gen, 59.33% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 38m 29s. Estimated total time: 18h 36m 8s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 12s, 500 more iterations: 3h 6m 1s.
[2025-11-13 09:01:27,504][__main__][INFO] - Starting iteration 162.
[2025-11-13 09:01:27,507][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:27,508][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:35,057][__main__][INFO] - Number of regex retries in iteration 162: 0
[2025-11-13 09:01:35,057][__main__][INFO] - agents played in iteration 162 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:01:35,629][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:35,665][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:35,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:35,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:35,732][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:35,732][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:36,537][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:36,833][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:37,159][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:37,487][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:37,813][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:01:38,139][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:01:38,465][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:01:38,790][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:01:39,118][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:01:39,445][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:01:39,771][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:01:40,096][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:01:40,422][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:01:40,747][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:01:41,073][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:01:41,405][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:01:41,730][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:01:42,055][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:01:42,382][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:01:42,713][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:01:43,045][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:01:43,373][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:01:43,699][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:01:44,024][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:01:44,349][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:01:44,674][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:01:45,000][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:45,325][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:45,651][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:45,976][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:46,301][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:46,626][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:46,954][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:47,703][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:48,440][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:48,442][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:48,444][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:49,451][__main__][INFO] - Iteration 163 took 21s (34.40% Gen, 61.00% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 19m 14s. Estimated total time: 18h 17m 15s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 34s, 500 more iterations: 3h 2m 52s.
[2025-11-13 09:01:49,454][__main__][INFO] - Starting iteration 163.
[2025-11-13 09:01:49,457][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:49,457][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:57,042][__main__][INFO] - Number of regex retries in iteration 163: 0
[2025-11-13 09:01:57,042][__main__][INFO] - agents played in iteration 163 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:01:57,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:57,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:57,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:57,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:57,623][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:57,623][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:58,410][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:58,707][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:59,034][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:59,358][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:59,684][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:00,010][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:00,336][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:00,660][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:00,991][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:01,312][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:01,637][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:01,962][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:02,293][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:02,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:02,950][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:03,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:03,604][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:03,930][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:04,254][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:04,579][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:04,908][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:05,235][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:05,559][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:05,886][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:06,211][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:06,541][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:06,867][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:07,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:07,524][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:07,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:08,174][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:08,499][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:08,822][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:09,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:10,267][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:10,268][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:10,270][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:11,240][__main__][INFO] - Iteration 164 took 21s (34.82% Gen, 60.72% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 10m 50s. Estimated total time: 18h 9m 12s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 18s, 500 more iterations: 3h 1m 32s.
[2025-11-13 09:02:11,242][__main__][INFO] - Starting iteration 164.
[2025-11-13 09:02:11,246][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:02:11,246][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:19,252][__main__][INFO] - Number of regex retries in iteration 164: 0
[2025-11-13 09:02:19,253][__main__][INFO] - agents played in iteration 164 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:02:19,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:19,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:19,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:19,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:19,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:19,857][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:20,623][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:02:20,920][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:02:21,246][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:02:21,575][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:02:21,899][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:22,226][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:22,552][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:22,877][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:23,201][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:23,525][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:23,849][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:24,175][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:24,501][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:24,828][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:25,154][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:25,478][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:25,807][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:26,136][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:26,463][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:26,789][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:27,114][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:27,437][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:27,761][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:28,087][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:28,409][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:28,733][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:29,063][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:29,388][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:29,714][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:30,038][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:30,368][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:30,697][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:31,024][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:31,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:32,525][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:32,527][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:32,528][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:33,510][__main__][INFO] - Iteration 165 took 22s (35.96% Gen, 59.63% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 34m 29s. Estimated total time: 18h 33m 13s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 6s, 500 more iterations: 3h 5m 32s.
[2025-11-13 09:02:33,512][__main__][INFO] - Starting iteration 165.
[2025-11-13 09:02:33,515][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:02:33,516][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:41,565][__main__][INFO] - Number of regex retries in iteration 165: 0
[2025-11-13 09:02:41,566][__main__][INFO] - agents played in iteration 165 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:02:42,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:42,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:42,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:42,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:42,170][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:42,170][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:42,938][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:02:43,234][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:02:43,559][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:02:43,884][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:02:44,212][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:44,536][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:44,861][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:45,185][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:45,510][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:45,835][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:46,159][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:46,483][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:46,808][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:47,133][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:47,460][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:47,784][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:48,109][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:48,434][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:48,758][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:49,083][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:49,409][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:49,737][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:50,072][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:50,397][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:50,720][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:51,045][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:51,378][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:51,706][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:52,031][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:52,356][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:52,684][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:53,008][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:53,333][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:54,086][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:54,824][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:54,825][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:54,827][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:55,826][__main__][INFO] - Iteration 166 took 22s (36.08% Gen, 59.44% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 36m 26s. Estimated total time: 18h 35m 33s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 11s, 500 more iterations: 3h 5m 55s.
[2025-11-13 09:02:55,828][__main__][INFO] - Starting iteration 166.
[2025-11-13 09:02:55,831][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:02:55,831][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:03,762][__main__][INFO] - Number of regex retries in iteration 166: 0
[2025-11-13 09:03:03,763][__main__][INFO] - agents played in iteration 166 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:03:04,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:04,295][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:04,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:04,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:04,364][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:04,364][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:05,134][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:05,430][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:05,757][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:06,087][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:06,414][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:06,744][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:07,071][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:07,396][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:07,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:08,048][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:08,386][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:08,708][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:09,033][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:09,360][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:09,695][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:10,012][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:10,336][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:10,660][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:10,990][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:11,307][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:11,630][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:11,954][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:12,278][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:12,602][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:12,926][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:13,251][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:03:13,583][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:03:13,909][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:03:14,237][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:03:14,562][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:03:14,889][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:03:15,214][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:03:15,539][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:03:16,294][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:03:17,027][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:03:17,028][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:03:17,030][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:18,017][__main__][INFO] - Iteration 167 took 22s (35.75% Gen, 59.79% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 53s. Estimated total time: 18h 29m 22s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 58s, 500 more iterations: 3h 4m 53s.
[2025-11-13 09:03:18,020][__main__][INFO] - Starting iteration 167.
[2025-11-13 09:03:18,023][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:03:18,023][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:25,589][__main__][INFO] - Number of regex retries in iteration 167: 0
[2025-11-13 09:03:25,589][__main__][INFO] - agents played in iteration 167 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:03:26,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:26,104][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:26,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:26,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:26,171][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:26,171][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:26,949][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:27,246][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:27,572][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:27,899][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:28,228][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:28,564][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:28,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:29,215][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:29,543][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:29,870][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:30,195][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:30,521][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:30,845][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:31,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:31,494][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:31,821][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:32,151][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:32,476][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:32,801][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:33,126][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:33,449][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:33,774][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:34,098][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:34,422][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:34,746][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:35,070][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:03:35,395][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:03:35,719][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:03:36,043][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:03:36,368][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:03:36,693][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:03:37,017][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:03:37,342][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:03:38,071][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:03:38,822][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:03:38,824][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:03:38,825][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:39,796][__main__][INFO] - Iteration 168 took 21s (34.75% Gen, 60.79% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 8m 50s. Estimated total time: 18h 8m 41s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 26s.
[2025-11-13 09:03:39,798][__main__][INFO] - Starting iteration 168.
[2025-11-13 09:03:39,801][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:03:39,802][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:47,561][__main__][INFO] - Number of regex retries in iteration 168: 0
[2025-11-13 09:03:47,562][__main__][INFO] - agents played in iteration 168 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:03:48,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:48,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:48,127][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:48,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:48,161][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:48,162][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:48,934][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:49,235][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:49,560][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:49,886][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:50,214][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:50,866][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:51,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:51,525][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:51,849][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:52,176][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:52,501][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:52,827][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:53,152][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:53,476][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:53,800][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:54,125][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:54,451][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:54,775][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:55,098][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:55,422][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:55,748][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:56,076][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:56,400][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:56,725][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:57,051][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:03:57,381][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:03:57,706][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:03:58,031][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:03:58,357][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:03:58,693][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:03:59,017][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:03:59,344][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:00,099][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:00,840][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:00,841][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:00,843][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:01,855][__main__][INFO] - Iteration 169 took 22s (35.18% Gen, 60.22% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 22m 32s. Estimated total time: 18h 22m 45s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 45s, 500 more iterations: 3h 3m 47s.
[2025-11-13 09:04:01,857][__main__][INFO] - Starting iteration 169.
[2025-11-13 09:04:01,861][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:04:01,861][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:09,750][__main__][INFO] - Number of regex retries in iteration 169: 0
[2025-11-13 09:04:09,751][__main__][INFO] - agents played in iteration 169 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:04:10,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:10,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:10,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:10,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:10,353][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:10,354][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:11,120][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:11,422][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:11,746][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:12,078][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:12,403][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:04:12,728][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:04:13,065][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:04:13,391][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:04:13,718][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:04:14,042][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:04:14,369][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:04:14,697][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:04:15,021][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:04:15,347][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:04:15,684][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:04:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:04:16,338][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:04:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:04:16,999][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:04:17,325][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:04:17,653][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:04:17,978][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:04:18,307][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:04:18,639][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:04:18,968][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:04:19,293][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:04:19,623][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:04:19,949][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:04:20,279][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:04:20,614][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:04:20,934][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:04:21,260][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:04:21,590][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:22,331][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:23,081][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:23,083][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:23,084][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:24,100][__main__][INFO] - Iteration 170 took 22s (35.47% Gen, 59.96% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 31m 25s. Estimated total time: 18h 32m 0s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 4s, 500 more iterations: 3h 5m 20s.
[2025-11-13 09:04:24,103][__main__][INFO] - Starting iteration 170.
[2025-11-13 09:04:24,106][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:04:24,106][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:32,121][__main__][INFO] - Number of regex retries in iteration 170: 0
[2025-11-13 09:04:32,122][__main__][INFO] - agents played in iteration 170 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:04:32,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:32,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:32,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:32,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:32,725][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:32,726][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:33,498][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:33,795][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:34,126][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:34,451][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:34,778][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:04:35,105][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:04:35,432][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:04:35,759][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:04:36,091][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:04:36,424][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:04:36,750][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:04:37,075][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:04:37,401][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:04:37,726][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:04:38,053][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:04:38,381][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:04:38,706][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:04:39,030][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:04:39,355][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:04:39,681][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:04:40,005][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:04:40,334][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:04:40,660][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:04:40,988][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:04:41,315][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:04:41,641][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:04:41,965][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:04:42,290][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:04:42,616][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:04:42,942][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:04:43,267][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:04:43,593][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:04:43,919][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:44,655][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:45,400][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:45,402][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:45,404][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:47,434][__main__][INFO] - Iteration 171 took 23s (34.36% Gen, 56.93% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 25m 28s. Estimated total time: 19h 26m 26s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 24s.
[2025-11-13 09:04:47,436][__main__][INFO] - Starting iteration 171.
[2025-11-13 09:04:47,439][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:04:47,439][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:56,069][__main__][INFO] - Number of regex retries in iteration 171: 0
[2025-11-13 09:04:56,069][__main__][INFO] - agents played in iteration 171 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:04:56,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:56,577][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:56,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:56,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:56,645][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:56,645][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:57,418][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:57,715][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:58,044][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:58,376][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:58,704][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:04:59,033][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:04:59,358][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:04:59,683][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:00,008][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:00,333][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:00,660][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:00,984][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:01,316][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:01,636][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:01,967][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:02,294][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:02,629][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:02,955][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:03,285][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:03,616][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:03,944][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:04,269][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:04,599][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:04,924][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:05,250][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:05,577][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:05,904][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:06,234][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:06,560][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:06,890][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:07,213][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:07,539][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:07,864][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:08,585][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:09,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:09,326][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:09,327][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:10,308][__main__][INFO] - Iteration 172 took 22s (37.73% Gen, 57.97% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 2m 7s. Estimated total time: 19h 3m 28s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 6s, 500 more iterations: 3h 10m 34s.
[2025-11-13 09:05:10,310][__main__][INFO] - Starting iteration 172.
[2025-11-13 09:05:10,313][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:10,314][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:05:18,520][__main__][INFO] - Number of regex retries in iteration 172: 0
[2025-11-13 09:05:18,521][__main__][INFO] - agents played in iteration 172 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:05:19,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:19,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:19,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:19,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:19,107][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:05:19,107][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:05:19,854][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:05:20,152][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:05:20,482][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:05:20,810][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:05:21,136][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:05:21,462][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:05:21,791][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:05:22,120][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:22,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:22,770][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:23,094][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:23,420][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:23,747][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:24,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:24,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:24,724][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:25,051][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:25,378][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:25,702][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:26,032][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:26,357][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:26,682][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:27,007][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:27,332][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:27,663][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:27,990][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:28,320][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:28,647][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:28,976][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:29,308][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:29,633][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:29,958][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:30,283][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:31,022][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:31,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:31,769][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:31,770][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:32,816][__main__][INFO] - Iteration 173 took 22s (36.47% Gen, 58.88% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 43m 26s. Estimated total time: 18h 45m 10s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 30s, 500 more iterations: 3h 7m 31s.
[2025-11-13 09:05:32,818][__main__][INFO] - Starting iteration 173.
[2025-11-13 09:05:32,821][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:32,822][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:05:41,469][__main__][INFO] - Number of regex retries in iteration 173: 0
[2025-11-13 09:05:41,469][__main__][INFO] - agents played in iteration 173 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:05:41,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:41,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,051][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:05:42,051][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:05:42,838][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:05:43,135][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:05:43,466][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:05:43,792][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:05:44,117][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:05:44,442][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:05:44,768][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:05:45,093][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:45,418][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:45,743][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:46,067][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:46,399][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:46,724][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:47,049][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:47,373][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:47,698][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:48,022][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:48,347][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:48,995][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:49,319][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:49,644][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:49,968][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:50,293][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:50,618][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:50,942][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:51,267][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:51,596][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:51,923][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:52,249][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:52,573][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:52,897][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:53,222][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:53,970][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:54,703][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:54,704][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:54,706][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:55,749][__main__][INFO] - Iteration 174 took 22s (37.71% Gen, 57.73% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 4m 19s. Estimated total time: 19h 6m 26s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 12s, 500 more iterations: 3h 11m 4s.
[2025-11-13 09:05:55,751][__main__][INFO] - Starting iteration 174.
[2025-11-13 09:05:55,754][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:55,755][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:04,075][__main__][INFO] - Number of regex retries in iteration 174: 0
[2025-11-13 09:06:04,076][__main__][INFO] - agents played in iteration 174 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:06:04,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,675][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:04,675][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:05,449][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:05,749][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:06,074][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:06,400][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:06,726][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:07,050][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:07,375][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:07,699][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:08,023][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:08,347][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:08,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:08,997][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:09,321][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:09,647][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:09,972][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:10,295][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:10,620][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:10,945][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:11,269][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:11,596][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:11,920][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:12,244][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:12,571][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:12,897][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:13,221][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:13,548][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:06:13,874][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:06:14,199][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:06:14,523][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:06:14,848][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:06:15,173][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:06:15,499][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:06:15,823][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:06:16,608][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:06:17,349][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:06:17,350][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:06:17,352][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:06:18,356][__main__][INFO] - Iteration 175 took 22s (36.81% Gen, 58.74% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 47m 38s. Estimated total time: 18h 50m 7s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 40s, 500 more iterations: 3h 8m 21s.
[2025-11-13 09:06:18,358][__main__][INFO] - Starting iteration 175.
[2025-11-13 09:06:18,361][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:06:18,361][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:27,093][__main__][INFO] - Number of regex retries in iteration 175: 0
[2025-11-13 09:06:27,093][__main__][INFO] - agents played in iteration 175 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:06:27,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,687][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:27,687][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:28,470][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:28,768][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:29,094][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:29,419][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:29,744][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:30,075][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:30,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:30,725][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:31,049][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:31,375][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:31,700][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:32,029][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:32,350][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:32,674][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:32,998][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:33,323][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:33,647][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:33,970][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:34,294][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:34,621][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:34,947][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:35,271][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:35,597][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:35,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:36,250][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:06:36,901][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:06:37,225][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:06:37,551][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:06:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:06:38,200][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:06:38,524][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:06:38,851][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:06:39,596][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:06:40,372][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:06:40,374][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:06:40,375][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:06:41,373][__main__][INFO] - Iteration 176 took 23s (37.94% Gen, 57.71% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 7m 47s. Estimated total time: 19h 10m 39s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 46s.
[2025-11-13 09:06:41,375][__main__][INFO] - Starting iteration 176.
[2025-11-13 09:06:41,378][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:06:41,378][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:50,058][__main__][INFO] - Number of regex retries in iteration 176: 0
[2025-11-13 09:06:50,059][__main__][INFO] - agents played in iteration 176 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:06:50,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,631][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,665][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:50,665][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:51,414][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:51,711][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:52,044][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:52,375][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:52,704][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:53,030][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:53,356][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:53,682][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:54,008][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:54,332][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:54,658][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:54,984][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:55,309][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:55,961][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:56,286][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:56,612][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:56,942][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:57,268][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:57,595][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:57,922][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:58,248][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:58,576][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:58,903][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:59,229][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:59,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:06:59,882][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:00,207][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:00,532][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:00,863][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:01,185][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:01,514][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:01,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:02,591][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:03,326][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:03,327][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:03,329][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:04,297][__main__][INFO] - Iteration 177 took 22s (37.87% Gen, 57.90% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 2m 45s. Estimated total time: 19h 6m 0s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 12s, 500 more iterations: 3h 11m 0s.
[2025-11-13 09:07:04,300][__main__][INFO] - Starting iteration 177.
[2025-11-13 09:07:04,303][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:04,304][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:13,050][__main__][INFO] - Number of regex retries in iteration 177: 0
[2025-11-13 09:07:13,050][__main__][INFO] - agents played in iteration 177 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:07:13,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,602][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,637][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:13,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:07:14,406][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:07:14,703][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:07:15,029][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:07:15,355][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:07:15,686][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:07:16,017][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:07:16,341][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:07:16,666][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:07:16,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:07:17,315][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:07:17,642][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:07:17,974][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:07:18,296][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:07:18,621][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:07:18,946][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:07:19,276][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:07:19,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:07:19,923][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:07:20,250][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:07:20,577][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:07:20,900][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:07:21,225][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:07:21,550][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:07:21,883][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:07:22,214][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:07:22,540][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:07:22,870][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:23,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:23,527][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:23,851][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:24,176][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:24,501][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:24,835][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:25,601][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:26,355][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:26,356][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:26,358][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:27,340][__main__][INFO] - Iteration 178 took 23s (37.96% Gen, 57.77% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 8m 15s. Estimated total time: 19h 11m 54s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 59s.
[2025-11-13 09:07:27,342][__main__][INFO] - Starting iteration 178.
[2025-11-13 09:07:27,345][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:27,346][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:36,112][__main__][INFO] - Number of regex retries in iteration 178: 0
[2025-11-13 09:07:36,112][__main__][INFO] - agents played in iteration 178 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:07:36,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,674][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,708][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:36,709][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:07:37,456][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:07:37,751][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:07:38,076][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:07:38,400][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:07:38,725][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:07:39,052][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:07:39,375][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:07:39,700][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:07:40,024][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:07:40,349][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:07:40,677][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:07:41,000][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:07:41,328][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:07:41,659][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:07:41,980][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:07:42,307][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:07:42,631][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:07:42,956][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:07:43,281][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:07:43,608][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:07:43,932][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:07:44,263][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:07:44,581][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:07:44,905][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:07:45,235][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:07:45,570][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:07:45,894][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:46,219][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:46,549][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:46,885][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:47,205][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:47,531][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:47,860][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:48,622][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:49,369][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:49,370][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:49,372][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:50,366][__main__][INFO] - Iteration 179 took 23s (38.08% Gen, 57.60% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 7m 3s. Estimated total time: 19h 11m 5s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 22s, 500 more iterations: 3h 11m 50s.
[2025-11-13 09:07:50,368][__main__][INFO] - Starting iteration 179.
[2025-11-13 09:07:50,372][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:50,372][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:58,781][__main__][INFO] - Number of regex retries in iteration 179: 0
[2025-11-13 09:07:58,782][__main__][INFO] - agents played in iteration 179 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:07:59,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,374][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,375][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:59,375][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:00,149][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:00,446][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:00,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:01,098][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:01,425][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:01,753][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:02,085][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:02,407][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:02,734][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:03,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:03,391][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:03,718][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:04,049][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:04,373][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:04,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:05,036][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:05,364][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:05,689][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:06,016][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:06,351][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:06,681][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:07,008][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:07,663][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:07,993][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:08,318][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:08,646][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:08,974][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:09,300][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:09,628][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:09,953][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:10,278][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:10,604][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:11,369][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:12,102][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:12,103][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:12,105][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:08:13,133][__main__][INFO] - Iteration 180 took 22s (36.95% Gen, 58.53% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 53m 40s. Estimated total time: 18h 58m 5s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 56s, 500 more iterations: 3h 9m 40s.
[2025-11-13 09:08:13,135][__main__][INFO] - Starting iteration 180.
[2025-11-13 09:08:13,138][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:08:13,139][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:08:21,171][__main__][INFO] - Number of regex retries in iteration 180: 0
[2025-11-13 09:08:21,172][__main__][INFO] - agents played in iteration 180 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:08:21,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:21,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:21,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:21,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:21,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:08:21,772][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:22,559][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:22,857][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:23,188][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:23,509][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:23,835][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:24,163][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:24,488][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:24,814][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:25,139][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:25,468][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:25,798][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:26,129][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:26,457][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:26,783][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:27,108][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:27,434][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:27,759][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:28,087][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:28,412][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:28,740][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:29,067][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:29,393][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:29,719][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:30,046][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:30,374][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:30,700][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:31,028][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:31,353][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:31,678][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:32,004][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:32,329][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:32,655][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:32,980][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:33,729][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:34,484][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:34,486][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:34,488][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:08:36,597][__main__][INFO] - Iteration 181 took 23s (34.24% Gen, 56.76% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 28m 9s. Estimated total time: 19h 32m 57s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 5s, 500 more iterations: 3h 15m 29s.
[2025-11-13 09:08:36,599][__main__][INFO] - Starting iteration 181.
[2025-11-13 09:08:36,602][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:08:36,602][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:08:45,679][__main__][INFO] - Number of regex retries in iteration 181: 0
[2025-11-13 09:08:45,680][__main__][INFO] - agents played in iteration 181 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:08:46,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,271][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:08:46,271][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:47,134][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:47,433][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:47,763][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:48,096][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:48,415][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:48,739][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:49,062][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:49,391][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:49,711][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:50,036][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:50,360][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:50,690][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:51,008][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:51,337][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:51,662][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:51,988][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:52,314][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:52,966][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:53,296][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:53,616][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:53,944][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:54,272][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:54,599][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:54,922][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:55,247][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:55,570][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:55,903][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:56,224][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:56,551][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:56,876][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:57,203][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:57,525][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:58,256][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:59,038][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:59,039][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:59,041][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:00,120][__main__][INFO] - Iteration 182 took 23s (38.59% Gen, 56.81% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 30m 46s. Estimated total time: 19h 35m 57s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 59s.
[2025-11-13 09:09:00,122][__main__][INFO] - Starting iteration 182.
[2025-11-13 09:09:00,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:00,126][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:08,941][__main__][INFO] - Number of regex retries in iteration 182: 0
[2025-11-13 09:09:08,942][__main__][INFO] - agents played in iteration 182 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:09:09,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:09,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:09,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:09,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:09,546][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:09,546][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:10,278][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:10,574][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:10,901][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:11,227][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:11,554][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:11,879][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:12,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:12,533][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:12,863][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:13,191][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:13,517][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:13,845][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:09:14,169][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:09:14,494][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:09:14,821][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:09:15,146][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:09:15,472][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:09:15,803][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:09:16,122][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:09:16,447][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:09:16,772][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:09:17,103][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:09:17,424][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:09:17,748][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:09:18,074][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:09:18,405][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:09:18,731][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:09:19,059][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:09:19,386][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:09:19,713][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:09:20,040][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:09:20,369][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:09:20,698][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:09:21,444][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:09:22,186][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:09:22,188][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:09:22,190][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:23,100][__main__][INFO] - Iteration 183 took 22s (38.37% Gen, 57.66% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 3m 12s. Estimated total time: 19h 8m 46s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 17s, 500 more iterations: 3h 11m 27s.
[2025-11-13 09:09:23,102][__main__][INFO] - Starting iteration 183.
[2025-11-13 09:09:23,104][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:23,104][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:32,096][__main__][INFO] - Number of regex retries in iteration 183: 0
[2025-11-13 09:09:32,097][__main__][INFO] - agents played in iteration 183 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:09:32,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:32,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:32,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:32,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:32,704][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:32,704][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:33,427][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:33,723][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:34,049][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:34,374][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:34,703][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:35,030][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:35,354][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:35,685][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:36,010][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:36,335][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:36,660][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:36,987][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:09:37,315][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:09:37,641][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:09:37,966][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:09:38,290][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:09:38,615][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:09:38,940][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:09:39,270][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:09:39,597][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:09:39,922][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:09:40,247][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:09:40,573][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:09:40,902][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:09:41,227][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:09:41,555][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:09:41,881][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:09:42,206][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:09:42,530][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:09:42,857][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:09:43,184][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:09:43,510][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:09:43,837][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:09:44,615][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:09:45,351][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:09:45,353][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:09:45,355][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:46,418][__main__][INFO] - Iteration 184 took 23s (38.57% Gen, 56.87% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 19m 45s. Estimated total time: 19h 25m 43s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 51s, 500 more iterations: 3h 14m 17s.
[2025-11-13 09:09:46,420][__main__][INFO] - Starting iteration 184.
[2025-11-13 09:09:46,423][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:46,423][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:55,018][__main__][INFO] - Number of regex retries in iteration 184: 0
[2025-11-13 09:09:55,019][__main__][INFO] - agents played in iteration 184 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:09:55,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:55,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:55,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:55,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:55,606][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:55,606][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:56,343][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:56,639][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:56,966][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:57,292][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:57,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:57,941][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:58,263][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:58,588][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:58,918][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:59,238][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:59,563][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:59,888][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:10:00,216][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:10:00,543][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:10:00,869][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:10:01,192][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:01,516][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:01,840][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:02,165][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:02,489][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:02,815][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:03,139][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:03,464][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:03,788][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:04,112][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:04,437][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:04,761][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:05,088][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:05,418][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:05,741][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:06,068][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:06,393][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:06,720][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:07,462][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:08,221][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:08,222][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:08,224][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:09,121][__main__][INFO] - Iteration 185 took 22s (37.87% Gen, 58.18% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 48m 36s. Estimated total time: 18h 54m 56s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 49s, 500 more iterations: 3h 9m 9s.
[2025-11-13 09:10:09,123][__main__][INFO] - Starting iteration 185.
[2025-11-13 09:10:09,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:09,126][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:10:17,889][__main__][INFO] - Number of regex retries in iteration 185: 0
[2025-11-13 09:10:17,890][__main__][INFO] - agents played in iteration 185 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:10:18,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,480][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:18,481][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:10:18,481][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:10:19,210][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:10:19,505][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:10:19,832][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:10:20,157][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:10:20,482][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:10:20,810][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:10:21,138][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:10:21,476][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:10:21,803][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:10:22,126][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:10:22,455][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:10:22,790][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:10:23,117][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:10:23,447][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:10:23,771][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:10:24,098][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:24,429][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:24,755][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:25,080][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:25,406][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:25,733][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:26,064][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:26,388][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:26,714][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:27,041][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:27,365][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:27,691][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:28,015][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:28,342][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:28,670][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:28,996][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:29,322][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:29,648][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:30,417][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:31,152][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:31,154][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:31,158][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:32,073][__main__][INFO] - Iteration 186 took 22s (38.19% Gen, 57.82% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 0m 43s. Estimated total time: 19h 7m 26s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 14s, 500 more iterations: 3h 11m 14s.
[2025-11-13 09:10:32,076][__main__][INFO] - Starting iteration 186.
[2025-11-13 09:10:32,078][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:32,079][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:10:40,812][__main__][INFO] - Number of regex retries in iteration 186: 0
[2025-11-13 09:10:40,813][__main__][INFO] - agents played in iteration 186 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:10:41,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:41,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:41,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:41,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:41,402][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:10:41,403][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:10:42,158][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:10:42,453][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:10:42,784][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:10:43,109][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:10:43,435][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:10:43,761][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:10:44,086][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:10:44,411][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:10:44,735][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:10:45,060][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:10:45,385][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:10:45,713][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:10:46,037][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:10:46,361][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:10:46,685][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:10:47,010][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:47,335][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:47,660][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:47,997][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:48,321][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:48,645][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:48,969][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:49,303][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:49,629][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:49,955][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:50,281][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:50,610][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:50,939][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:51,263][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:51,589][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:51,917][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:52,243][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:52,568][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:53,348][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:54,059][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:54,061][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:54,063][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:55,323][__main__][INFO] - Iteration 187 took 23s (37.57% Gen, 57.00% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 15m 9s. Estimated total time: 19h 22m 16s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 42s.
[2025-11-13 09:10:55,325][__main__][INFO] - Starting iteration 187.
[2025-11-13 09:10:55,327][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:55,328][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:03,034][__main__][INFO] - Number of regex retries in iteration 187: 0
[2025-11-13 09:11:03,035][__main__][INFO] - agents played in iteration 187 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:11:03,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:03,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:03,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:03,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:03,613][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:03,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:04,349][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:04,646][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:04,971][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:05,297][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:05,624][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:05,955][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:06,284][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:06,614][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:06,941][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:07,265][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:07,596][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:07,925][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:08,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:08,583][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:08,912][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:09,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:09,560][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:09,891][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:10,225][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:10,547][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:10,878][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:11,202][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:11,531][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:11,852][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:12,178][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:12,505][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:12,834][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:13,163][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:13,489][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:13,817][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:14,144][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:14,474][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:11:14,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:11:15,553][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:11:16,258][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:16,260][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:16,261][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:11:17,182][__main__][INFO] - Iteration 188 took 21s (35.26% Gen, 60.52% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 5m 19s. Estimated total time: 18h 12m 48s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 25s, 500 more iterations: 3h 2m 8s.
[2025-11-13 09:11:17,184][__main__][INFO] - Starting iteration 188.
[2025-11-13 09:11:17,187][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:11:17,187][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:25,836][__main__][INFO] - Number of regex retries in iteration 188: 0
[2025-11-13 09:11:25,837][__main__][INFO] - agents played in iteration 188 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:11:26,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:26,344][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:26,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:26,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:26,410][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:26,411][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:27,143][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:27,439][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:27,765][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:28,090][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:28,416][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:28,744][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:29,071][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:29,396][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:29,720][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:30,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:30,370][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:30,694][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:31,019][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:31,343][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:31,667][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:31,992][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:32,316][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:32,642][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:32,966][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:33,291][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:33,942][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:34,267][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:34,592][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:34,916][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:35,240][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:35,565][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:35,891][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:36,218][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:36,545][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:36,873][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:37,198][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:11:37,528][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:11:38,287][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:11:39,007][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:39,010][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:39,012][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:11:39,916][__main__][INFO] - Iteration 189 took 22s (38.05% Gen, 57.97% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 48m 38s. Estimated total time: 18h 56m 30s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 53s, 500 more iterations: 3h 9m 25s.
[2025-11-13 09:11:39,918][__main__][INFO] - Starting iteration 189.
[2025-11-13 09:11:39,921][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:11:39,921][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:48,596][__main__][INFO] - Number of regex retries in iteration 189: 0
[2025-11-13 09:11:48,597][__main__][INFO] - agents played in iteration 189 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:11:49,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:49,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:49,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:49,166][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:49,167][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:49,167][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:49,920][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:50,215][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:50,867][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:51,204][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:51,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:51,853][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:52,177][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:52,511][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:52,836][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:53,161][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:53,486][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:53,815][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:54,140][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:54,466][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:54,791][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:55,114][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:55,439][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:55,763][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:56,087][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:56,411][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:56,736][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:57,060][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:57,385][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:57,716][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:58,042][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:58,368][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:58,693][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:59,019][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:59,345][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:59,670][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:59,995][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:12:00,322][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:12:01,064][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:12:01,791][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:12:01,792][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:12:01,794][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:02,706][__main__][INFO] - Iteration 190 took 22s (38.07% Gen, 57.92% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 51m 5s. Estimated total time: 18h 59m 19s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 58s, 500 more iterations: 3h 9m 53s.
[2025-11-13 09:12:02,708][__main__][INFO] - Starting iteration 190.
[2025-11-13 09:12:02,711][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:12:02,711][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:11,431][__main__][INFO] - Number of regex retries in iteration 190: 0
[2025-11-13 09:12:11,431][__main__][INFO] - agents played in iteration 190 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:12:11,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:11,939][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:11,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:12,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:12,006][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:12,006][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:12,747][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:13,043][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:13,370][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:12:13,697][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:12:14,022][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:12:14,353][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:12:14,680][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:12:15,005][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:12:15,332][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:12:15,656][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:12:15,982][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:12:16,306][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:12:16,632][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:12:16,957][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:12:17,289][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:12:17,613][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:12:17,938][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:12:18,264][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:12:18,592][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:12:18,917][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:12:19,244][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:12:19,570][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:12:19,895][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:12:20,223][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:12:20,551][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:12:20,876][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:12:21,202][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:12:21,529][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:12:21,858][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:12:22,184][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:12:22,511][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:12:22,839][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:12:23,170][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:12:23,920][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:12:24,653][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:12:24,656][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:12:24,657][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:26,428][__main__][INFO] - Iteration 191 took 23s (36.76% Gen, 55.76% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 37m 16s. Estimated total time: 19h 45m 53s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 38s.
[2025-11-13 09:12:26,430][__main__][INFO] - Starting iteration 191.
[2025-11-13 09:12:26,432][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:12:26,433][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:35,125][__main__][INFO] - Number of regex retries in iteration 191: 0
[2025-11-13 09:12:35,126][__main__][INFO] - agents played in iteration 191 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:12:35,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:35,631][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:35,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:35,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:35,697][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:35,698][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:36,453][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:36,752][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:37,084][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:12:37,412][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:12:37,739][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:12:38,068][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:12:38,394][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:12:38,719][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:12:39,045][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:12:39,371][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:12:39,695][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:12:40,019][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:12:40,343][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:12:40,668][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:12:40,992][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:12:41,317][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:12:41,641][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:12:41,965][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:12:42,290][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:12:42,614][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:12:42,941][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:12:43,271][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:12:43,595][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:12:43,923][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:12:44,249][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:12:44,575][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:12:44,902][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:12:45,227][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:12:45,553][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:12:45,883][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:12:46,208][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:12:46,534][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:12:46,861][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:12:47,625][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:12:48,325][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:12:48,327][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:12:48,328][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:49,348][__main__][INFO] - Iteration 192 took 22s (37.93% Gen, 57.61% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 56m 47s. Estimated total time: 19h 5m 48s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 58s.
[2025-11-13 09:12:49,349][__main__][INFO] - Starting iteration 192.
[2025-11-13 09:12:49,352][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:12:49,352][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:57,936][__main__][INFO] - Number of regex retries in iteration 192: 0
[2025-11-13 09:12:57,936][__main__][INFO] - agents played in iteration 192 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:12:58,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:58,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:58,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:58,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:58,504][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:58,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:59,251][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:59,548][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:59,873][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:13:00,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:13:00,531][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:13:00,856][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:13:01,181][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:13:01,504][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:13:01,829][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:13:02,154][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:13:02,478][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:13:02,802][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:03,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:03,456][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:03,782][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:04,106][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:04,433][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:04,758][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:05,082][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:05,407][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:05,733][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:06,058][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:06,382][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:06,710][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:07,036][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:07,361][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:07,686][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:08,012][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:08,338][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:08,669][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:08,994][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:09,321][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:09,651][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:10,412][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:11,147][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:11,149][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:11,150][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:12,110][__main__][INFO] - Iteration 193 took 22s (37.72% Gen, 58.06% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 48m 33s. Estimated total time: 18h 57m 56s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 55s, 500 more iterations: 3h 9m 39s.
[2025-11-13 09:13:12,112][__main__][INFO] - Starting iteration 193.
[2025-11-13 09:13:12,115][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:12,116][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:13:20,510][__main__][INFO] - Number of regex retries in iteration 193: 0
[2025-11-13 09:13:20,511][__main__][INFO] - agents played in iteration 193 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:13:20,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:21,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:21,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:21,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:21,099][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:13:21,099][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:13:21,852][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:13:22,147][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:13:22,478][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:13:22,804][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:13:23,131][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:13:23,456][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:13:23,780][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:13:24,112][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:13:24,436][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:13:24,765][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:13:25,091][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:13:25,419][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:25,746][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:26,070][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:26,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:26,721][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:27,047][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:27,377][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:27,704][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:28,029][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:28,356][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:28,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:29,006][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:29,333][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:29,661][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:29,984][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:30,311][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:30,641][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:30,968][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:31,294][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:31,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:31,955][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:32,273][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:33,021][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:33,766][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:33,768][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:33,770][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:34,686][__main__][INFO] - Iteration 194 took 22s (37.19% Gen, 58.74% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 38m 49s. Estimated total time: 18h 48m 35s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 37s, 500 more iterations: 3h 8m 5s.
[2025-11-13 09:13:34,688][__main__][INFO] - Starting iteration 194.
[2025-11-13 09:13:34,691][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:34,691][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:13:43,167][__main__][INFO] - Number of regex retries in iteration 194: 0
[2025-11-13 09:13:43,168][__main__][INFO] - agents played in iteration 194 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:13:43,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:43,711][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:43,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:43,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:43,778][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:13:43,779][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:13:44,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:13:44,813][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:13:45,143][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:13:45,466][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:13:45,795][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:13:46,123][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:13:46,449][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:13:46,777][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:13:47,102][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:13:47,429][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:13:47,756][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:13:48,084][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:48,408][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:48,735][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:49,063][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:49,387][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:49,712][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:50,038][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:50,363][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:50,693][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:51,020][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:51,347][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:51,676][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:52,001][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:52,326][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:52,654][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:52,980][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:53,308][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:53,634][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:53,962][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:54,288][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:54,613][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:54,939][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:55,721][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:56,438][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:56,441][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:56,447][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:57,376][__main__][INFO] - Iteration 195 took 22s (37.36% Gen, 58.53% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 44m 10s. Estimated total time: 18h 54m 19s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 48s, 500 more iterations: 3h 9m 3s.
[2025-11-13 09:13:57,378][__main__][INFO] - Starting iteration 195.
[2025-11-13 09:13:57,381][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:57,382][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:06,146][__main__][INFO] - Number of regex retries in iteration 195: 0
[2025-11-13 09:14:06,147][__main__][INFO] - agents played in iteration 195 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:14:06,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:06,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:06,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:06,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:06,722][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:06,722][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:07,439][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:07,735][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:08,060][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:08,390][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:08,715][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:09,045][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:09,372][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:09,697][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:10,024][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:10,350][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:10,674][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:10,999][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:11,329][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:11,657][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:11,983][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:12,310][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:12,647][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:12,972][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:13,297][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:13,622][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:13,951][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:14,276][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:14,604][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:14:14,928][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:14:15,255][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:14:15,580][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:14:15,905][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:14:16,232][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:14:16,559][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:14:16,885][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:14:17,211][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:14:17,536][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:14:17,862][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:14:18,624][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:14:19,344][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:19,345][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:19,347][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:20,262][__main__][INFO] - Iteration 196 took 22s (38.31% Gen, 57.69% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 53m 33s. Estimated total time: 19h 4m 5s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 8s, 500 more iterations: 3h 10m 40s.
[2025-11-13 09:14:20,264][__main__][INFO] - Starting iteration 196.
[2025-11-13 09:14:20,267][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:14:20,268][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:28,832][__main__][INFO] - Number of regex retries in iteration 196: 0
[2025-11-13 09:14:28,833][__main__][INFO] - agents played in iteration 196 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:14:29,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:29,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:29,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:29,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:29,419][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:29,419][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:30,154][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:30,449][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:30,775][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:31,099][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:31,423][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:31,750][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:32,074][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:32,399][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:32,724][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:33,051][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:33,376][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:33,700][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:34,025][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:34,350][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:34,675][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:34,999][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:35,327][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:35,656][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:35,983][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:36,310][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:36,637][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:36,968][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:37,296][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:14:37,623][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:14:37,948][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:14:38,274][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:14:38,599][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:14:38,924][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:14:39,251][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:14:39,580][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:14:39,905][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:14:40,232][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:14:40,557][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:14:41,323][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:14:42,042][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:42,044][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:42,045][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:42,992][__main__][INFO] - Iteration 197 took 22s (37.69% Gen, 58.14% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 45m 23s. Estimated total time: 18h 56m 17s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 52s, 500 more iterations: 3h 9m 22s.
[2025-11-13 09:14:42,994][__main__][INFO] - Starting iteration 197.
[2025-11-13 09:14:42,997][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:14:42,997][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:51,425][__main__][INFO] - Number of regex retries in iteration 197: 0
[2025-11-13 09:14:51,425][__main__][INFO] - agents played in iteration 197 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:14:51,905][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:51,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:51,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:52,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:52,005][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:52,006][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:52,700][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:52,995][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:53,319][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:53,644][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:53,971][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:54,295][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:54,622][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:54,959][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:55,609][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:55,934][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:56,263][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:56,587][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:56,912][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:57,238][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:57,567][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:57,892][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:58,217][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:58,545][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:58,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:59,199][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:59,525][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:59,854][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:15:00,178][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:15:00,503][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:15:00,829][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:15:01,160][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:15:01,481][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:15:01,807][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:15:02,133][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:15:02,459][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:15:02,789][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:15:03,115][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:15:03,884][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:15:04,575][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:04,577][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:04,579][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:05,533][__main__][INFO] - Iteration 198 took 22s (37.40% Gen, 58.37% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 35m 32s. Estimated total time: 18h 46m 49s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 48s.
[2025-11-13 09:15:05,534][__main__][INFO] - Starting iteration 198.
[2025-11-13 09:15:05,537][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:15:05,537][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:14,093][__main__][INFO] - Number of regex retries in iteration 198: 0
[2025-11-13 09:15:14,094][__main__][INFO] - agents played in iteration 198 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:15:14,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:14,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:14,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:14,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:14,673][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:14,674][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:15,355][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:15,650][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:15:15,975][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:15:16,300][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:15:16,625][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:15:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:15:17,275][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:15:17,598][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:15:17,923][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:15:18,246][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:15:18,573][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:15:18,897][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:15:19,221][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:15:19,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:15:19,874][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:15:20,200][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:15:20,526][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:15:20,851][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:15:21,176][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:15:21,501][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:15:21,827][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:15:22,151][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:15:22,476][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:15:22,802][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:15:23,128][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:15:23,461][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:15:23,782][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:15:24,107][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:15:24,435][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:15:24,762][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:15:25,089][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:15:25,415][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:15:25,740][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:15:26,523][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:15:27,230][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:27,232][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:27,233][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:28,141][__main__][INFO] - Iteration 199 took 22s (37.85% Gen, 58.13% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 38m 33s. Estimated total time: 18h 50m 13s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 40s, 500 more iterations: 3h 8m 22s.
[2025-11-13 09:15:28,143][__main__][INFO] - Starting iteration 199.
[2025-11-13 09:15:28,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:15:28,146][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:35,914][__main__][INFO] - Number of regex retries in iteration 199: 0
[2025-11-13 09:15:35,915][__main__][INFO] - agents played in iteration 199 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:15:36,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:36,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:36,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:36,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:36,487][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:36,487][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:37,246][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:37,542][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:15:37,868][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:15:38,194][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:15:38,519][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:15:38,843][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:15:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:15:39,497][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:15:39,824][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:15:40,150][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:15:40,484][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:15:40,801][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:15:41,127][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:15:41,454][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:15:41,790][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:15:42,109][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:15:42,434][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:15:42,760][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:15:43,095][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:15:43,421][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:15:43,746][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:15:44,075][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:15:44,399][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:15:44,726][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:15:45,054][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:15:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:15:45,708][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:15:46,033][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:15:46,359][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:15:46,684][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:15:47,010][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:15:47,336][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:15:47,662][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:15:48,429][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:15:49,125][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:49,126][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:49,128][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:50,034][__main__][INFO] - Iteration 200 took 21s (35.49% Gen, 60.36% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 2m 27s. Estimated total time: 18h 14m 28s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 28s, 500 more iterations: 3h 2m 24s.
[2025-11-13 09:15:50,036][__main__][INFO] - Starting iteration 200.
[2025-11-13 09:15:50,039][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:15:50,039][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:58,300][__main__][INFO] - Number of regex retries in iteration 200: 0
[2025-11-13 09:15:58,301][__main__][INFO] - agents played in iteration 200 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:15:58,782][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:58,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:58,849][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:58,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:58,897][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:58,898][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:59,656][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:59,952][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:16:00,278][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:16:00,604][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:16:00,927][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:16:01,252][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:16:01,576][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:16:01,900][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:16:02,228][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:16:02,553][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:16:02,878][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:16:03,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:16:03,527][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:16:03,851][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:16:04,180][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:16:04,504][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:16:04,829][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:16:05,156][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:05,481][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:05,808][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:06,133][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:06,459][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:06,785][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:07,111][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:07,437][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:07,770][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:08,095][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:08,420][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:08,746][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:09,071][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:09,396][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:09,722][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:10,048][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:10,810][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:11,525][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:11,526][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:11,528][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:13,422][__main__][INFO] - Iteration 201 took 23s (35.33% Gen, 56.57% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 16m 49s. Estimated total time: 19h 29m 14s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 52s.
[2025-11-13 09:16:13,425][__main__][INFO] - Starting iteration 201.
[2025-11-13 09:16:13,427][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:13,428][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:22,213][__main__][INFO] - Number of regex retries in iteration 201: 0
[2025-11-13 09:16:22,214][__main__][INFO] - agents played in iteration 201 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:16:22,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:22,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:22,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:22,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:22,798][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:22,798][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:23,537][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:16:23,832][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:16:24,160][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:16:24,484][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:16:24,809][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:16:25,133][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:16:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:16:25,785][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:16:26,113][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:16:26,438][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:16:26,763][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:16:27,087][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:16:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:16:27,737][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:16:28,063][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:16:28,389][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:16:28,716][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:16:29,042][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:29,368][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:29,695][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:30,021][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:30,346][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:30,671][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:30,996][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:31,321][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:31,648][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:31,973][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:32,298][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:32,624][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:32,950][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:33,280][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:33,609][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:33,934][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:34,697][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:35,399][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:35,400][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:35,402][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:36,354][__main__][INFO] - Iteration 202 took 22s (38.32% Gen, 57.52% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 53m 35s. Estimated total time: 19h 6m 23s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 12s, 500 more iterations: 3h 11m 3s.
[2025-11-13 09:16:36,356][__main__][INFO] - Starting iteration 202.
[2025-11-13 09:16:36,359][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:36,359][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:44,881][__main__][INFO] - Number of regex retries in iteration 202: 0
[2025-11-13 09:16:44,882][__main__][INFO] - agents played in iteration 202 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:16:45,357][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:45,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:45,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:45,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:45,459][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:45,459][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:46,196][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:16:46,492][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:16:46,817][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:16:47,142][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:16:47,468][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:16:47,793][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:16:48,120][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:16:48,446][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:16:48,775][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:16:49,100][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:16:49,425][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:16:49,751][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:16:50,082][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:16:50,409][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:16:50,735][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:16:51,061][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:16:51,393][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:16:51,721][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:52,046][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:52,371][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:52,700][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:53,029][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:53,354][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:53,679][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:54,007][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:54,334][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:54,660][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:54,986][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:55,313][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:55,965][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:56,291][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:56,618][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:57,382][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:58,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:58,106][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:58,107][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:59,036][__main__][INFO] - Iteration 203 took 22s (37.58% Gen, 58.32% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 40m 44s. Estimated total time: 18h 53m 55s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 47s, 500 more iterations: 3h 8m 59s.
[2025-11-13 09:16:59,038][__main__][INFO] - Starting iteration 203.
[2025-11-13 09:16:59,041][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:59,042][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:17:07,543][__main__][INFO] - Number of regex retries in iteration 203: 0
[2025-11-13 09:17:07,544][__main__][INFO] - agents played in iteration 203 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:17:08,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:08,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:08,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:08,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:08,135][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:17:08,136][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:17:08,902][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:17:09,198][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:09,524][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:09,849][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:10,175][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:10,503][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:10,832][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:11,158][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:11,483][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:11,808][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:12,135][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:12,462][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:12,787][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:13,436][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:13,762][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:14,087][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:14,417][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:14,743][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:17:15,068][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:17:15,394][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:17:15,719][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:17:16,045][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:17:16,369][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:17:16,695][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:17:17,020][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:17:17,345][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:17:17,670][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:17:17,995][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:17:18,320][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:17:18,646][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:17:18,972][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:17:19,297][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:17:20,069][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:17:20,801][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:20,803][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:20,805][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:21,773][__main__][INFO] - Iteration 204 took 22s (37.40% Gen, 58.33% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 43m 5s. Estimated total time: 18h 56m 38s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 53s, 500 more iterations: 3h 9m 26s.
[2025-11-13 09:17:21,775][__main__][INFO] - Starting iteration 204.
[2025-11-13 09:17:21,779][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:17:21,780][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:17:30,178][__main__][INFO] - Number of regex retries in iteration 204: 0
[2025-11-13 09:17:30,179][__main__][INFO] - agents played in iteration 204 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:17:30,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:30,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:30,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:30,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:30,765][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:17:30,765][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:17:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:17:31,796][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:32,123][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:32,450][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:32,777][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:33,103][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:33,429][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:33,757][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:34,085][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:34,413][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:34,744][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:35,066][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:35,394][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:35,724][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:36,050][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:36,376][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:36,704][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:37,031][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:37,359][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:17:37,685][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:17:38,016][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:17:38,342][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:17:38,667][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:17:38,996][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:17:39,323][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:17:39,648][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:17:39,974][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:17:40,301][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:17:40,628][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:17:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:17:41,285][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:17:41,611][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:17:41,937][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:17:42,687][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:17:43,421][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:43,422][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:43,424][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:44,440][__main__][INFO] - Iteration 205 took 22s (37.06% Gen, 58.45% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 39m 10s. Estimated total time: 18h 53m 5s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 46s, 500 more iterations: 3h 8m 50s.
[2025-11-13 09:17:44,442][__main__][INFO] - Starting iteration 205.
[2025-11-13 09:17:44,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:17:44,446][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:17:49,206][mllm.models.large_language_model_local][WARNING] - Response <B><tool_call>
user
<A>
<OK> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 09:17:52,703][__main__][INFO] - Number of regex retries in iteration 205: 1
[2025-11-13 09:17:52,704][__main__][INFO] - agents played in iteration 205 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:17:53,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:53,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:53,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:53,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:53,286][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:17:53,287][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:17:54,009][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:17:54,304][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:54,630][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:54,957][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:55,607][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:55,934][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:56,258][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:56,584][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:56,910][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:57,237][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:57,563][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:57,888][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:58,216][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:58,545][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:58,871][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:59,195][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:59,524][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:59,846][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:00,170][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:00,495][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:00,831][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:01,150][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:01,476][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:01,803][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:02,136][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:02,455][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:02,779][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:03,104][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:03,436][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:03,755][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:04,081][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:04,409][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:18:05,193][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:18:05,950][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:05,952][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:05,953][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:06,942][__main__][INFO] - Iteration 206 took 22s (36.70% Gen, 58.89% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 30m 37s. Estimated total time: 18h 44m 55s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 29s, 500 more iterations: 3h 7m 29s.
[2025-11-13 09:18:06,945][__main__][INFO] - Starting iteration 206.
[2025-11-13 09:18:06,948][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:18:06,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:15,299][__main__][INFO] - Number of regex retries in iteration 206: 0
[2025-11-13 09:18:15,300][__main__][INFO] - agents played in iteration 206 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:18:15,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:15,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:15,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:15,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:15,878][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:15,879][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:16,637][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:18:16,936][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:18:17,261][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:18:17,587][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:18:17,913][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:18:18,242][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:18:18,567][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:18:18,897][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:18:19,228][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:18:19,549][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:18:19,874][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:18:20,200][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:18:20,526][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:18:20,855][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:18:21,181][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:18:21,507][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:18:21,839][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:18:22,157][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:18:22,482][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:22,807][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:23,139][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:23,458][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:23,782][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:24,107][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:24,438][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:24,756][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:25,081][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:25,410][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:25,742][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:26,066][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:26,392][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:26,717][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:27,048][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:18:27,811][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:18:28,562][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:28,564][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:28,566][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:29,568][__main__][INFO] - Iteration 207 took 22s (36.92% Gen, 58.64% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 36m 21s. Estimated total time: 18h 51m 1s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 42s, 500 more iterations: 3h 8m 30s.
[2025-11-13 09:18:29,570][__main__][INFO] - Starting iteration 207.
[2025-11-13 09:18:29,573][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:18:29,574][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:37,940][__main__][INFO] - Number of regex retries in iteration 207: 0
[2025-11-13 09:18:37,940][__main__][INFO] - agents played in iteration 207 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:18:38,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:38,465][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:38,498][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:38,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:38,532][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:38,532][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:39,301][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:18:39,597][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:18:39,923][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:18:40,255][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:18:40,579][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:18:40,904][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:18:41,228][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:18:41,553][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:18:41,877][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:18:42,203][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:18:42,529][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:18:42,855][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:18:43,180][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:18:43,505][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:18:43,829][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:18:44,157][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:18:44,481][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:18:44,807][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:18:45,133][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:45,458][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:45,783][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:46,107][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:46,433][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:46,758][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:47,084][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:47,409][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:47,735][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:48,063][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:48,386][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:48,712][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:49,036][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:49,362][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:49,687][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:18:50,473][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:18:51,228][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:51,230][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:51,231][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:52,227][__main__][INFO] - Iteration 208 took 22s (36.93% Gen, 58.67% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 37m 40s. Estimated total time: 18h 52m 44s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 45s, 500 more iterations: 3h 8m 47s.
[2025-11-13 09:18:52,229][__main__][INFO] - Starting iteration 208.
[2025-11-13 09:18:52,232][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:18:52,232][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:00,635][__main__][INFO] - Number of regex retries in iteration 208: 0
[2025-11-13 09:19:00,636][__main__][INFO] - agents played in iteration 208 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:19:01,138][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:01,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:01,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:01,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:01,238][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:01,238][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:02,009][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:02,305][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:02,631][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:02,957][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:03,284][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:03,610][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:03,936][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:04,262][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:04,587][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:04,913][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:05,242][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:19:05,568][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:19:05,894][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:19:06,219][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:19:06,550][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:19:06,877][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:19:07,204][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:19:07,541][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:19:07,860][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:19:08,186][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:19:08,512][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:19:08,850][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:19:09,168][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:19:09,493][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:19:09,820][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:19:10,153][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:19:10,473][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:19:10,800][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:19:11,128][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:19:11,461][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:19:11,781][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:19:12,107][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:19:12,432][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:13,204][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:13,950][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:13,952][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:13,953][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:14,934][__main__][INFO] - Iteration 209 took 22s (37.02% Gen, 58.66% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 39m 41s. Estimated total time: 18h 55m 8s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 50s, 500 more iterations: 3h 9m 11s.
[2025-11-13 09:19:14,936][__main__][INFO] - Starting iteration 209.
[2025-11-13 09:19:14,939][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:19:14,939][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:23,191][__main__][INFO] - Number of regex retries in iteration 209: 0
[2025-11-13 09:19:23,191][__main__][INFO] - agents played in iteration 209 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:19:23,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:23,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:23,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:23,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:23,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:23,779][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:24,522][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:24,832][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:25,159][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:25,487][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:25,816][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:26,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:26,473][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:26,797][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:27,124][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:27,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:27,775][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:19:28,105][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:19:28,439][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:19:28,759][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:19:29,083][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:19:29,409][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:19:29,741][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:19:30,060][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:19:30,385][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:19:30,711][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:19:31,036][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:19:31,361][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:19:31,687][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:19:32,013][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:19:32,339][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:19:32,664][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:19:32,990][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:19:33,315][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:19:33,641][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:19:33,968][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:19:34,301][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:19:34,625][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:19:34,950][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:35,711][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:36,478][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:36,479][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:36,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:37,467][__main__][INFO] - Iteration 210 took 22s (36.63% Gen, 58.99% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 30m 39s. Estimated total time: 18h 46m 28s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 32s, 500 more iterations: 3h 7m 44s.
[2025-11-13 09:19:37,470][__main__][INFO] - Starting iteration 210.
[2025-11-13 09:19:37,473][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:19:37,473][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:45,363][__main__][INFO] - Number of regex retries in iteration 210: 0
[2025-11-13 09:19:45,364][__main__][INFO] - agents played in iteration 210 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:19:45,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:45,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:45,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:45,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:45,960][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:45,961][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:46,714][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:47,010][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:47,344][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:47,669][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:47,995][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:48,322][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:48,971][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:49,297][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:49,623][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:49,948][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:19:50,274][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:19:50,602][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:19:50,928][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:19:51,258][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:19:51,582][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:19:51,909][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:19:52,241][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:19:52,562][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:19:52,893][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:19:53,218][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:19:53,553][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:19:53,874][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:19:54,199][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:19:54,525][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:19:54,857][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:19:55,179][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:19:55,505][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:19:55,835][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:19:56,160][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:19:56,488][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:19:56,815][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:19:57,139][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:57,883][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:58,618][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:58,619][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:58,621][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:00,792][__main__][INFO] - Iteration 211 took 23s (33.83% Gen, 56.85% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 9m 48s. Estimated total time: 19h 26m 0s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 20s.
[2025-11-13 09:20:00,794][__main__][INFO] - Starting iteration 211.
[2025-11-13 09:20:00,797][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:00,798][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:09,228][__main__][INFO] - Number of regex retries in iteration 211: 0
[2025-11-13 09:20:09,229][__main__][INFO] - agents played in iteration 211 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:20:09,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:09,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:09,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:09,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:09,820][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:09,820][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:10,574][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:20:10,869][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:20:11,195][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:20:11,522][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:20:11,846][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:20:12,173][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:20:12,502][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:20:12,827][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:20:13,154][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:20:13,480][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:20:13,810][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:14,134][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:14,459][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:20:14,785][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:20:15,112][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:20:15,437][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:20:15,762][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:20:16,089][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:20:16,414][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:20:16,739][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:20:17,065][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:20:17,390][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:20:17,718][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:20:18,044][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:20:18,370][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:20:18,695][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:20:19,020][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:20:19,347][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:20:19,674][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:20:19,999][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:20:20,324][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:20:20,650][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:20:20,974][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:20:21,687][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:20:22,421][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:22,423][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:22,425][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:23,404][__main__][INFO] - Iteration 212 took 22s (37.29% Gen, 58.37% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 33m 48s. Estimated total time: 18h 50m 23s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 40s, 500 more iterations: 3h 8m 23s.
[2025-11-13 09:20:23,406][__main__][INFO] - Starting iteration 212.
[2025-11-13 09:20:23,409][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:23,410][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:31,704][__main__][INFO] - Number of regex retries in iteration 212: 0
[2025-11-13 09:20:31,704][__main__][INFO] - agents played in iteration 212 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:20:32,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:32,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:32,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:32,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:32,304][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:32,304][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:33,073][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:20:33,370][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:20:33,696][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:20:34,021][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:20:34,348][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:20:34,675][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:20:34,999][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:20:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:20:35,650][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:20:35,975][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:20:36,302][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:36,632][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:36,959][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:20:37,290][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:20:37,616][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:20:37,942][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:20:38,269][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:20:38,595][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:20:38,922][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:20:39,248][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:20:39,576][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:20:39,903][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:20:40,229][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:20:40,554][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:20:40,881][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:20:41,205][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:20:41,533][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:20:41,859][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:20:42,184][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:20:42,508][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:20:42,836][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:20:43,161][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:20:43,487][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:20:44,188][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:20:44,937][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:44,939][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:44,940][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:45,926][__main__][INFO] - Iteration 213 took 22s (36.83% Gen, 58.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 28m 54s. Estimated total time: 18h 45m 51s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 31s, 500 more iterations: 3h 7m 38s.
[2025-11-13 09:20:45,928][__main__][INFO] - Starting iteration 213.
[2025-11-13 09:20:45,932][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:45,932][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:54,545][__main__][INFO] - Number of regex retries in iteration 213: 0
[2025-11-13 09:20:54,546][__main__][INFO] - agents played in iteration 213 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:20:55,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:55,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:55,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:55,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:55,153][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:55,153][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:55,942][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:20:56,239][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:20:56,574][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:20:56,894][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:20:57,221][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:20:57,547][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:20:57,881][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:20:58,208][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:20:58,536][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:20:58,865][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:20:59,193][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:59,519][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:59,845][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:00,170][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:00,496][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:00,823][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:01,148][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:01,475][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:01,801][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:02,127][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:02,454][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:02,779][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:03,105][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:03,438][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:03,766][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:04,094][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:04,425][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:04,758][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:05,087][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:05,414][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:05,740][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:06,068][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:06,395][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:07,140][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:07,866][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:07,868][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:07,869][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:08,842][__main__][INFO] - Iteration 214 took 22s (37.60% Gen, 58.15% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 48m 12s. Estimated total time: 19h 5m 32s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 55s.
[2025-11-13 09:21:08,844][__main__][INFO] - Starting iteration 214.
[2025-11-13 09:21:08,847][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:08,847][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:17,047][__main__][INFO] - Number of regex retries in iteration 214: 0
[2025-11-13 09:21:17,047][__main__][INFO] - agents played in iteration 214 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:21:17,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:17,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:17,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:17,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:17,629][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:17,630][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:18,406][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:21:18,704][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:21:19,029][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:21:19,354][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:21:19,680][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:21:20,006][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:21:20,331][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:21:20,663][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:21:20,989][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:21:21,315][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:21:21,641][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:21:21,968][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:21:22,294][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:22,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:22,950][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:23,274][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:23,600][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:23,927][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:24,258][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:24,580][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:24,904][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:25,230][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:25,562][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:25,884][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:26,209][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:26,535][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:26,862][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:27,186][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:27,512][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:27,840][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:28,165][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:28,492][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:28,818][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:29,531][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:30,289][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:30,291][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:30,292][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:31,356][__main__][INFO] - Iteration 215 took 22s (36.43% Gen, 58.84% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 27m 46s. Estimated total time: 18h 45m 28s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 30s, 500 more iterations: 3h 7m 34s.
[2025-11-13 09:21:31,358][__main__][INFO] - Starting iteration 215.
[2025-11-13 09:21:31,362][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:31,362][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:39,565][__main__][INFO] - Number of regex retries in iteration 215: 0
[2025-11-13 09:21:39,566][__main__][INFO] - agents played in iteration 215 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:21:40,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:40,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:40,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:40,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:40,185][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:40,185][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:40,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:21:41,258][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:21:41,581][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:21:41,906][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:21:42,231][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:21:42,558][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:21:42,887][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:21:43,213][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:21:43,539][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:21:43,865][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:21:44,194][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:21:44,519][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:21:44,846][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:45,172][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:45,500][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:45,824][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:46,150][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:46,476][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:46,801][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:47,126][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:47,453][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:47,778][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:48,103][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:48,430][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:48,755][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:49,080][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:49,405][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:49,729][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:50,054][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:50,380][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:50,705][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:51,036][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:51,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:52,119][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:52,876][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:52,877][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:52,879][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:53,880][__main__][INFO] - Iteration 216 took 22s (36.43% Gen, 59.12% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 27m 52s. Estimated total time: 18h 45m 57s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 31s, 500 more iterations: 3h 7m 39s.
[2025-11-13 09:21:53,882][__main__][INFO] - Starting iteration 216.
[2025-11-13 09:21:53,885][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:53,886][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:02,506][__main__][INFO] - Number of regex retries in iteration 216: 0
[2025-11-13 09:22:02,507][__main__][INFO] - agents played in iteration 216 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:22:03,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:03,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:03,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:03,140][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:03,141][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:03,141][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:03,914][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:04,211][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:04,538][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:04,864][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:05,192][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:05,517][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:05,842][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:06,169][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:06,494][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:06,819][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:07,144][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:07,476][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:07,801][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:08,127][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:08,460][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:08,784][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:09,110][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:09,440][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:09,765][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:10,090][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:10,416][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:22:10,743][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:22:11,069][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:22:11,395][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:22:11,719][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:22:12,048][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:22:12,374][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:22:12,700][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:22:13,026][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:22:13,355][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:22:13,680][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:22:14,006][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:22:14,334][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:22:15,085][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:22:15,828][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:22:15,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:22:15,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:22:16,800][__main__][INFO] - Iteration 217 took 22s (37.62% Gen, 58.15% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 47m 19s. Estimated total time: 19h 5m 47s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 57s.
[2025-11-13 09:22:16,802][__main__][INFO] - Starting iteration 217.
[2025-11-13 09:22:16,805][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:22:16,806][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:25,024][__main__][INFO] - Number of regex retries in iteration 217: 0
[2025-11-13 09:22:25,024][__main__][INFO] - agents played in iteration 217 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:22:25,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:25,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:25,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:25,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:25,617][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:25,618][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:26,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:26,698][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:27,025][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:27,352][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:27,685][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:28,006][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:28,332][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:28,659][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:28,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:29,309][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:29,635][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:29,962][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:30,294][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:30,940][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:31,267][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:31,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:31,919][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:32,244][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:32,571][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:32,896][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:22:33,222][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:22:33,549][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:22:33,874][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:22:34,203][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:22:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:22:34,857][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:22:35,183][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:22:35,508][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:22:35,836][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:22:36,160][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:22:36,485][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:22:36,812][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:22:37,562][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:22:38,338][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:22:38,339][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:22:38,341][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:22:39,341][__main__][INFO] - Iteration 218 took 22s (36.46% Gen, 59.09% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 27m 58s. Estimated total time: 18h 46m 48s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 48s.
[2025-11-13 09:22:39,343][__main__][INFO] - Starting iteration 218.
[2025-11-13 09:22:39,346][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:22:39,347][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:47,672][__main__][INFO] - Number of regex retries in iteration 218: 0
[2025-11-13 09:22:47,673][__main__][INFO] - agents played in iteration 218 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:22:48,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:48,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:48,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:48,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:48,271][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:48,271][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:49,024][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:49,647][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:49,981][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:50,302][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:50,628][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:50,954][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:51,287][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:51,606][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:51,934][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:52,262][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:52,594][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:52,912][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:53,241][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:53,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:53,897][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:54,219][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:54,546][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:54,872][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:55,199][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:55,523][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:22:55,848][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:22:56,174][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:22:56,501][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:22:56,823][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:22:57,150][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:22:57,475][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:22:57,803][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:22:58,128][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:22:58,452][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:22:58,776][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:22:59,102][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:22:59,427][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:00,141][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:00,879][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:00,880][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:00,882][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:01,884][__main__][INFO] - Iteration 219 took 22s (36.95% Gen, 58.60% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 27m 42s. Estimated total time: 18h 46m 55s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 49s.
[2025-11-13 09:23:01,886][__main__][INFO] - Starting iteration 219.
[2025-11-13 09:23:01,889][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:23:01,890][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:10,151][__main__][INFO] - Number of regex retries in iteration 219: 0
[2025-11-13 09:23:10,151][__main__][INFO] - agents played in iteration 219 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:23:10,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:10,677][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:10,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:10,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:10,744][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:10,745][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:11,526][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:23:11,823][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:23:12,148][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:23:12,480][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:23:12,807][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:23:13,131][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:23:13,457][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:23:13,784][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:23:14,109][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:23:14,435][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:23:14,761][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:23:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:23:15,412][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:23:15,741][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:23:16,074][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:23:16,398][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:23:16,726][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:23:17,055][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:23:17,387][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:23:17,706][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:23:18,033][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:23:18,358][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:23:18,688][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:23:19,020][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:23:19,346][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:23:19,671][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:23:20,001][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:23:20,321][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:23:20,648][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:23:20,976][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:23:21,306][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:23:21,629][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:23:21,954][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:22,639][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:23,385][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:23,386][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:23,388][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:24,419][__main__][INFO] - Iteration 220 took 22s (36.67% Gen, 58.75% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 26m 57s. Estimated total time: 18h 46m 33s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 45s.
[2025-11-13 09:23:24,421][__main__][INFO] - Starting iteration 220.
[2025-11-13 09:23:24,425][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:23:24,425][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:32,758][__main__][INFO] - Number of regex retries in iteration 220: 0
[2025-11-13 09:23:32,758][__main__][INFO] - agents played in iteration 220 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:23:33,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:33,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:33,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:33,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:33,378][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:33,378][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:34,171][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:23:34,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:23:34,796][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:23:35,121][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:23:35,447][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:23:35,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:23:36,097][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:23:36,425][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:23:36,750][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:23:37,075][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:23:37,405][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:23:37,732][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:23:38,059][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:23:38,388][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:23:38,713][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:23:39,038][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:23:39,365][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:23:39,690][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:23:40,018][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:23:40,340][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:23:40,664][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:23:40,988][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:23:41,315][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:23:41,641][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:23:41,969][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:23:42,295][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:23:42,621][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:23:42,947][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:23:43,273][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:23:43,601][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:23:43,928][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:23:44,256][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:23:44,584][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:45,347][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:46,086][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:46,087][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:46,089][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:48,084][__main__][INFO] - Iteration 221 took 23s (35.22% Gen, 56.34% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 23m 1s. Estimated total time: 19h 43m 0s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 26s, 500 more iterations: 3h 17m 10s.
[2025-11-13 09:23:48,086][__main__][INFO] - Starting iteration 221.
[2025-11-13 09:23:48,090][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:23:48,091][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:56,907][__main__][INFO] - Number of regex retries in iteration 221: 0
[2025-11-13 09:23:56,908][__main__][INFO] - agents played in iteration 221 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:23:57,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:57,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:57,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:57,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:57,497][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:57,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:58,298][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:23:58,597][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:23:58,923][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:23:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:23:59,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:23:59,905][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:00,230][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:00,556][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:00,885][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:01,212][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:01,541][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:01,869][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:02,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:02,516][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:02,846][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:03,172][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:03,498][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:03,826][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:04,153][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:04,478][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:04,802][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:05,129][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:05,456][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:05,782][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:06,107][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:06,437][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:06,763][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:07,087][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:07,417][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:07,745][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:08,073][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:08,397][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:08,722][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:09,493][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:10,251][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:10,252][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:10,254][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:11,250][__main__][INFO] - Iteration 222 took 23s (38.06% Gen, 57.62% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 57m 41s. Estimated total time: 19h 18m 3s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 0s.
[2025-11-13 09:24:11,252][__main__][INFO] - Starting iteration 222.
[2025-11-13 09:24:11,256][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:24:11,257][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:20,015][__main__][INFO] - Number of regex retries in iteration 222: 0
[2025-11-13 09:24:20,016][__main__][INFO] - agents played in iteration 222 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:24:20,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:20,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:20,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:20,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:20,616][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:20,616][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:21,378][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:24:21,675][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:24:22,000][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:24:22,327][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:24:22,651][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:24:22,982][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:23,308][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:23,635][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:23,963][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:24,301][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:24,619][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:24,946][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:25,272][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:25,603][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:25,922][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:26,245][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:26,573][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:26,906][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:27,232][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:27,565][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:27,893][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:28,220][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:28,547][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:28,878][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:29,202][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:29,527][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:29,854][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:30,180][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:30,508][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:30,839][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:31,170][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:31,497][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:31,822][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:32,556][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:33,291][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:33,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:33,294][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:34,401][__main__][INFO] - Iteration 223 took 23s (37.85% Gen, 57.37% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 56m 31s. Estimated total time: 19h 17m 17s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 34s, 500 more iterations: 3h 12m 52s.
[2025-11-13 09:24:34,403][__main__][INFO] - Starting iteration 223.
[2025-11-13 09:24:34,406][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:24:34,407][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:43,079][__main__][INFO] - Number of regex retries in iteration 223: 0
[2025-11-13 09:24:43,080][__main__][INFO] - agents played in iteration 223 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:24:43,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:43,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:43,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:43,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:43,670][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:43,670][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:44,451][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:24:44,748][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:24:45,074][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:24:45,399][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:24:45,724][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:24:46,054][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:46,382][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:46,712][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:47,037][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:47,363][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:47,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:48,013][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:48,337][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:48,663][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:48,988][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:49,313][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:49,639][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:49,966][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:50,290][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:50,621][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:50,941][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:51,269][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:51,594][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:51,929][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:52,253][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:52,580][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:52,906][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:53,237][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:53,560][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:53,890][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:54,552][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:54,879][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:55,624][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:56,383][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:56,385][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:56,387][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:57,385][__main__][INFO] - Iteration 224 took 22s (37.74% Gen, 57.91% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 47m 51s. Estimated total time: 19h 8m 59s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 17s, 500 more iterations: 3h 11m 29s.
[2025-11-13 09:24:57,387][__main__][INFO] - Starting iteration 224.
[2025-11-13 09:24:57,390][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:24:57,391][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:06,487][__main__][INFO] - Number of regex retries in iteration 224: 0
[2025-11-13 09:25:06,487][__main__][INFO] - agents played in iteration 224 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:25:07,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:07,038][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:07,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:07,105][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:07,106][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:07,106][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:07,857][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:08,153][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:08,479][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:08,806][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:09,131][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:09,457][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:09,782][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:10,108][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:10,433][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:10,757][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:11,083][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:25:11,408][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:25:11,734][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:25:12,061][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:25:12,385][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:25:12,710][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:25:13,035][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:25:13,359][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:25:13,685][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:25:14,009][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:25:14,339][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:25:14,664][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:25:14,990][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:25:15,316][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:25:15,641][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:25:15,966][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:25:16,290][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:25:16,615][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:25:16,939][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:25:17,263][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:25:17,588][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:25:17,914][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:25:18,240][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:25:18,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:25:19,739][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:25:19,741][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:25:19,742][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:20,723][__main__][INFO] - Iteration 225 took 23s (38.99% Gen, 56.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 5m 9s. Estimated total time: 19h 26m 41s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 26s.
[2025-11-13 09:25:20,725][__main__][INFO] - Starting iteration 225.
[2025-11-13 09:25:20,728][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:25:20,729][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:30,033][__main__][INFO] - Number of regex retries in iteration 225: 0
[2025-11-13 09:25:30,034][__main__][INFO] - agents played in iteration 225 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:25:30,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:30,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:30,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:30,651][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:30,652][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:30,652][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:31,376][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:31,672][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:31,999][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:32,325][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:32,650][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:32,975][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:33,302][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:33,628][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:33,952][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:34,278][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:34,603][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:25:34,927][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:25:35,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:25:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:25:35,903][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:25:36,228][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:25:36,553][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:25:36,878][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:25:37,206][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:25:37,532][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:25:37,857][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:25:38,181][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:25:38,508][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:25:38,833][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:25:39,157][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:25:39,483][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:25:39,807][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:25:40,133][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:25:40,457][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:25:40,781][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:25:41,116][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:25:41,444][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:25:41,771][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:25:42,529][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:25:43,248][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:25:43,249][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:25:43,251][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:44,237][__main__][INFO] - Iteration 226 took 23s (39.58% Gen, 56.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 13m 33s. Estimated total time: 19h 35m 29s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 54s.
[2025-11-13 09:25:44,239][__main__][INFO] - Starting iteration 226.
[2025-11-13 09:25:44,243][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:25:44,243][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:53,161][__main__][INFO] - Number of regex retries in iteration 226: 0
[2025-11-13 09:25:53,162][__main__][INFO] - agents played in iteration 226 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:25:53,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:53,697][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:53,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:53,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:53,767][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:53,767][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:54,531][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:54,827][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:55,154][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:55,478][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:55,806][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:56,131][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:56,455][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:56,780][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:57,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:57,435][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:57,760][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:25:58,086][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:25:58,414][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:25:58,742][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:25:59,069][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:25:59,406][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:25:59,733][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:00,058][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:00,385][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:00,710][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:01,036][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:01,361][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:01,685][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:02,010][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:02,336][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:02,662][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:02,988][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:03,315][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:03,641][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:03,966][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:04,290][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:04,621][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:04,944][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:05,648][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:06,401][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:06,402][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:06,404][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:07,406][__main__][INFO] - Iteration 227 took 23s (38.50% Gen, 57.17% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 55m 54s. Estimated total time: 19h 18m 13s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 2s.
[2025-11-13 09:26:07,408][__main__][INFO] - Starting iteration 227.
[2025-11-13 09:26:07,412][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:26:07,412][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:26:16,180][__main__][INFO] - Number of regex retries in iteration 227: 0
[2025-11-13 09:26:16,181][__main__][INFO] - agents played in iteration 227 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:26:16,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:16,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:16,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:16,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:16,784][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:26:16,784][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:26:17,539][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:26:17,835][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:26:18,161][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:26:18,488][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:26:18,822][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:26:19,146][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:26:19,477][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:26:19,802][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:26:20,127][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:26:20,453][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:26:20,778][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:26:21,107][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:26:21,438][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:26:21,761][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:26:22,092][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:26:22,421][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:26:22,752][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:23,077][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:23,402][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:23,728][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:24,053][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:24,383][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:24,714][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:25,040][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:25,368][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:25,692][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:26,021][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:26,344][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:26,667][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:26,992][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:27,318][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:27,644][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:27,967][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:28,649][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:29,388][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:29,390][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:29,391][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:30,390][__main__][INFO] - Iteration 228 took 22s (38.16% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 46m 15s. Estimated total time: 19h 8m 57s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 17s, 500 more iterations: 3h 11m 29s.
[2025-11-13 09:26:30,392][__main__][INFO] - Starting iteration 228.
[2025-11-13 09:26:30,395][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:26:30,395][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:26:39,434][__main__][INFO] - Number of regex retries in iteration 228: 0
[2025-11-13 09:26:39,435][__main__][INFO] - agents played in iteration 228 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:26:39,927][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:39,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:39,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:40,028][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:40,028][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:26:40,029][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:26:40,783][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:26:41,080][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:26:41,418][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:26:41,747][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:26:42,072][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:26:42,399][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:26:42,726][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:26:43,053][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:26:43,379][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:26:43,705][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:26:44,032][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:26:44,357][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:26:44,680][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:26:45,004][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:26:45,331][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:26:45,656][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:26:45,979][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:46,304][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:46,631][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:46,958][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:47,283][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:47,608][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:47,934][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:48,258][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:48,585][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:48,909][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:49,239][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:49,567][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:49,897][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:50,221][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:50,545][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:50,869][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:51,194][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:51,890][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:52,628][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:52,630][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:52,632][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:53,601][__main__][INFO] - Iteration 229 took 23s (38.95% Gen, 56.87% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 57m 16s. Estimated total time: 19h 20m 20s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 40s, 500 more iterations: 3h 13m 23s.
[2025-11-13 09:26:53,603][__main__][INFO] - Starting iteration 229.
[2025-11-13 09:26:53,606][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:26:53,607][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:02,496][__main__][INFO] - Number of regex retries in iteration 229: 0
[2025-11-13 09:27:02,496][__main__][INFO] - agents played in iteration 229 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:27:02,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:03,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:03,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:03,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:03,073][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:03,074][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:03,822][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:04,119][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:04,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:04,771][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:05,106][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:05,432][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:05,757][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:06,083][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:06,410][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:06,739][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:07,066][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:07,392][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:07,718][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:08,044][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:08,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:08,692][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:09,017][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:09,342][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:09,667][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:27:09,994][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:27:10,317][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:27:10,647][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:27:10,970][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:27:11,294][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:27:11,619][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:27:11,946][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:27:12,271][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:27:12,596][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:27:12,922][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:27:13,245][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:27:13,570][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:27:13,895][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:27:14,223][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:27:14,910][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:27:15,662][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:27:15,663][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:27:15,665][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:27:16,734][__main__][INFO] - Iteration 230 took 23s (38.43% Gen, 56.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 52m 57s. Estimated total time: 19h 16m 25s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 32s, 500 more iterations: 3h 12m 44s.
[2025-11-13 09:27:16,736][__main__][INFO] - Starting iteration 230.
[2025-11-13 09:27:16,739][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:27:16,740][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:24,948][__main__][INFO] - Number of regex retries in iteration 230: 0
[2025-11-13 09:27:24,948][__main__][INFO] - agents played in iteration 230 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:27:25,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:25,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:25,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:25,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:25,531][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:25,532][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:26,302][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:26,598][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:26,930][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:27,256][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:27,587][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:27,914][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:28,247][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:28,568][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:28,898][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:29,224][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:29,554][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:29,880][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:30,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:30,529][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:30,858][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:31,179][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:31,510][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:31,835][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:32,164][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:27:32,492][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:27:32,821][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:27:33,149][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:27:33,472][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:27:33,797][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:27:34,124][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:27:34,450][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:27:34,777][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:27:35,108][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:27:35,433][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:27:35,760][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:27:36,084][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:27:36,421][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:27:36,747][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:27:37,448][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:27:38,181][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:27:38,182][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:27:38,184][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:27:40,102][__main__][INFO] - Iteration 231 took 23s (35.13% Gen, 56.65% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 4m 21s. Estimated total time: 19h 28m 12s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 42s.
[2025-11-13 09:27:40,104][__main__][INFO] - Starting iteration 231.
[2025-11-13 09:27:40,108][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:27:40,108][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:49,266][__main__][INFO] - Number of regex retries in iteration 231: 0
[2025-11-13 09:27:49,267][__main__][INFO] - agents played in iteration 231 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:27:49,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:49,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:49,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:49,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:49,847][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:49,847][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:50,607][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:50,911][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:51,241][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:51,565][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:51,890][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:52,543][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:52,870][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:53,197][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:53,523][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:53,848][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:54,173][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:54,500][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:54,826][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:55,151][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:55,480][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:55,804][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:56,137][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:56,464][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:27:56,790][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:27:57,117][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:27:57,446][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:27:57,771][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:27:58,095][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:27:58,420][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:27:58,748][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:27:59,073][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:27:59,397][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:27:59,722][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:00,049][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:00,375][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:00,699][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:01,024][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:01,730][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:02,473][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:02,474][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:02,476][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:03,458][__main__][INFO] - Iteration 232 took 23s (39.22% Gen, 56.57% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 3m 19s. Estimated total time: 19h 27m 33s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 35s.
[2025-11-13 09:28:03,460][__main__][INFO] - Starting iteration 232.
[2025-11-13 09:28:03,463][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:03,464][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:12,092][__main__][INFO] - Number of regex retries in iteration 232: 0
[2025-11-13 09:28:12,093][__main__][INFO] - agents played in iteration 232 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:28:12,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:12,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:12,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:12,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:12,676][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:12,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:13,437][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:28:13,733][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:28:14,059][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:28:14,385][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:28:14,710][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:28:15,037][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:28:15,362][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:28:15,687][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:28:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:28:16,340][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:28:16,667][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:28:16,994][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:28:17,320][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:28:17,646][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:28:17,971][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:28:18,297][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:28:18,623][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:28:18,951][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:28:19,277][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:28:19,604][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:28:19,930][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:28:20,254][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:28:20,580][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:28:20,910][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:28:21,240][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:28:21,566][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:28:21,897][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:28:22,225][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:28:22,552][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:22,880][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:23,204][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:23,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:23,857][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:24,540][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:25,287][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:25,289][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:25,291][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:26,305][__main__][INFO] - Iteration 233 took 22s (37.77% Gen, 57.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 37m 28s. Estimated total time: 19h 2m 6s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 4s, 500 more iterations: 3h 10m 21s.
[2025-11-13 09:28:26,306][__main__][INFO] - Starting iteration 233.
[2025-11-13 09:28:26,310][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:26,311][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:35,281][__main__][INFO] - Number of regex retries in iteration 233: 0
[2025-11-13 09:28:35,282][__main__][INFO] - agents played in iteration 233 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:28:35,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:35,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:35,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:35,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:35,880][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:35,881][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:36,645][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:28:36,949][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:28:37,268][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:28:37,593][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:28:37,919][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:28:38,249][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:28:38,567][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:28:38,891][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:28:39,217][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:28:39,543][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:28:39,869][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:28:40,196][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:28:40,523][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:28:40,852][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:28:41,177][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:28:41,504][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:28:41,832][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:28:42,158][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:28:42,485][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:28:42,811][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:28:43,135][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:28:43,466][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:28:43,785][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:28:44,111][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:28:44,437][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:28:44,764][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:28:45,089][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:28:45,417][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:28:45,745][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:46,071][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:46,396][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:46,721][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:47,048][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:47,733][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:48,464][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:48,465][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:48,467][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:49,446][__main__][INFO] - Iteration 234 took 23s (38.78% Gen, 56.99% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 51m 48s. Estimated total time: 19h 16m 49s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 48s.
[2025-11-13 09:28:49,448][__main__][INFO] - Starting iteration 234.
[2025-11-13 09:28:49,451][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:49,451][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:57,773][__main__][INFO] - Number of regex retries in iteration 234: 0
[2025-11-13 09:28:57,773][__main__][INFO] - agents played in iteration 234 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:28:58,264][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:58,297][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:58,331][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:58,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:58,365][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:58,366][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:59,116][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:28:59,416][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:28:59,741][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:00,066][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:00,393][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:00,721][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:01,049][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:01,375][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:01,700][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:02,032][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:02,357][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:02,682][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:03,007][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:03,342][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:03,668][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:03,995][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:04,322][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:04,646][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:04,971][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:05,295][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:05,953][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:06,277][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:06,601][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:06,926][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:29:07,251][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:29:07,576][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:29:07,901][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:29:08,226][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:29:08,557][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:29:08,880][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:29:09,206][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:29:09,529][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:29:10,213][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:29:10,955][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:10,957][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:10,959][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:11,969][__main__][INFO] - Iteration 235 took 22s (36.95% Gen, 58.55% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 20m 35s. Estimated total time: 18h 45m 58s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 31s, 500 more iterations: 3h 7m 39s.
[2025-11-13 09:29:11,971][__main__][INFO] - Starting iteration 235.
[2025-11-13 09:29:11,975][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:29:11,975][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:20,293][__main__][INFO] - Number of regex retries in iteration 235: 0
[2025-11-13 09:29:20,294][__main__][INFO] - agents played in iteration 235 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:29:20,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:20,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:20,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:20,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:20,878][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:20,878][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:21,634][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:29:21,939][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:29:22,265][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:22,590][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:22,914][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:23,254][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:23,580][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:23,908][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:24,239][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:24,565][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:24,890][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:25,215][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:25,541][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:25,869][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:26,197][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:26,528][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:26,858][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:27,189][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:27,515][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:27,841][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:28,169][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:28,494][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:28,819][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:29,144][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:29,469][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:29:29,794][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:29:30,121][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:29:30,445][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:29:30,768][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:29:31,093][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:29:31,419][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:29:31,742][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:29:32,067][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:29:32,753][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:29:33,500][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:33,501][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:33,503][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:34,474][__main__][INFO] - Iteration 236 took 22s (36.97% Gen, 58.71% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 19m 13s. Estimated total time: 18h 44m 59s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 29s, 500 more iterations: 3h 7m 29s.
[2025-11-13 09:29:34,475][__main__][INFO] - Starting iteration 236.
[2025-11-13 09:29:34,479][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:29:34,479][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:42,813][__main__][INFO] - Number of regex retries in iteration 236: 0
[2025-11-13 09:29:42,814][__main__][INFO] - agents played in iteration 236 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:29:43,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:43,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:43,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:43,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:43,397][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:43,398][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:44,163][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:29:44,462][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:29:44,788][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:45,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:45,440][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:45,769][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:46,094][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:46,421][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:46,745][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:47,075][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:47,401][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:47,728][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:48,058][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:48,390][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:48,718][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:49,043][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:49,368][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:49,693][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:50,018][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:50,343][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:50,669][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:50,996][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:51,323][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:51,653][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:51,979][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:29:52,305][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:29:52,633][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:29:52,958][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:29:53,282][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:29:53,606][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:29:53,930][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:29:54,255][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:29:54,579][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:29:55,285][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:29:56,029][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:56,031][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:56,032][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:57,039][__main__][INFO] - Iteration 237 took 22s (36.94% Gen, 58.59% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 21m 53s. Estimated total time: 18h 48m 2s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 36s, 500 more iterations: 3h 8m 0s.
[2025-11-13 09:29:57,041][__main__][INFO] - Starting iteration 237.
[2025-11-13 09:29:57,045][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:29:57,045][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:30:05,734][__main__][INFO] - Number of regex retries in iteration 237: 0
[2025-11-13 09:30:05,734][__main__][INFO] - agents played in iteration 237 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:30:06,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:06,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:06,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:06,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:06,323][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:30:06,324][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:30:07,085][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:30:07,381][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:30:07,706][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:30:08,032][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:30:08,357][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:30:08,683][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:30:09,009][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:30:09,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:30:09,659][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:30:09,986][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:30:10,312][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:30:10,638][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:30:10,969][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:30:11,291][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:30:11,616][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:30:11,941][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:30:12,273][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:30:12,594][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:30:12,921][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:30:13,247][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:30:13,572][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:30:13,899][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:30:14,227][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:30:14,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:30:14,887][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:30:15,206][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:30:15,531][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:30:15,857][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:30:16,192][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:30:16,513][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:30:16,838][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:30:17,162][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:30:17,492][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:30:18,198][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:30:18,947][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:18,949][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:18,951][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:19,931][__main__][INFO] - Iteration 238 took 22s (37.96% Gen, 57.74% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 37m 50s. Estimated total time: 19h 4m 21s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 8s, 500 more iterations: 3h 10m 43s.
[2025-11-13 09:30:19,933][__main__][INFO] - Starting iteration 238.
[2025-11-13 09:30:19,936][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:30:19,937][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:30:28,257][__main__][INFO] - Number of regex retries in iteration 238: 0
[2025-11-13 09:30:28,258][__main__][INFO] - agents played in iteration 238 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:30:28,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:28,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:28,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:28,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:28,881][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:30:28,881][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:30:29,657][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:30:29,953][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:30:30,280][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:30:30,608][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:30:30,935][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:30:31,272][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:30:31,597][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:30:31,922][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:30:32,248][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:30:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:30:32,903][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:30:33,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:30:33,553][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:30:33,881][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:30:34,206][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:30:34,532][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:30:34,858][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:30:35,186][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:30:35,512][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:30:35,837][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:30:36,163][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:30:36,491][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:30:36,818][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:30:37,146][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:30:37,471][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:30:37,805][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:30:38,125][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:30:38,451][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:30:38,776][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:30:39,104][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:30:39,430][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:30:39,758][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:30:40,083][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:30:40,792][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:30:41,540][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:41,541][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:41,543][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:42,530][__main__][INFO] - Iteration 239 took 22s (36.83% Gen, 58.80% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 22m 48s. Estimated total time: 18h 49m 41s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 39s, 500 more iterations: 3h 8m 16s.
[2025-11-13 09:30:42,532][__main__][INFO] - Starting iteration 239.
[2025-11-13 09:30:42,535][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:30:42,535][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:30:51,014][__main__][INFO] - Number of regex retries in iteration 239: 0
[2025-11-13 09:30:51,014][__main__][INFO] - agents played in iteration 239 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:30:51,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:51,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:51,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:51,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:51,607][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:30:51,608][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:30:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:30:52,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:30:53,019][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:30:53,345][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:30:53,670][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:30:53,996][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:30:54,322][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:30:54,646][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:30:54,974][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:30:55,297][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:30:55,624][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:30:55,949][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:30:56,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:30:56,601][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:30:56,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:30:57,251][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:30:57,584][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:30:57,901][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:30:58,228][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:30:58,556][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:30:58,887][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:30:59,209][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:30:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:30:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:00,196][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:00,519][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:31:00,844][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:31:01,171][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:31:01,502][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:31:01,824][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:31:02,155][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:31:02,481][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:31:02,807][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:31:03,572][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:31:04,334][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:04,336][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:04,337][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:05,304][__main__][INFO] - Iteration 240 took 22s (37.24% Gen, 58.51% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 31m 12s. Estimated total time: 18h 58m 28s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 56s, 500 more iterations: 3h 9m 44s.
[2025-11-13 09:31:05,306][__main__][INFO] - Starting iteration 240.
[2025-11-13 09:31:05,309][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:31:05,310][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:13,680][__main__][INFO] - Number of regex retries in iteration 240: 0
[2025-11-13 09:31:13,680][__main__][INFO] - agents played in iteration 240 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:31:14,166][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:14,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:14,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:14,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:14,267][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:14,268][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:15,056][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:31:15,354][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:31:15,682][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:31:16,008][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:31:16,337][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:31:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:31:16,991][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:31:17,316][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:31:17,642][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:31:17,968][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:31:18,295][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:31:18,619][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:31:18,946][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:31:19,271][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:31:19,599][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:31:19,923][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:31:20,248][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:31:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:31:20,900][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:31:21,226][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:31:21,551][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:31:21,878][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:31:22,205][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:31:22,534][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:22,860][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:23,185][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:31:23,511][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:31:23,837][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:31:24,168][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:31:24,495][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:31:24,823][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:31:25,149][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:31:25,474][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:31:26,239][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:31:26,993][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:26,995][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:26,996][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:28,953][__main__][INFO] - Iteration 241 took 23s (35.40% Gen, 56.32% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 14m 35s. Estimated total time: 19h 42m 15s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 24s, 500 more iterations: 3h 17m 2s.
[2025-11-13 09:31:28,956][__main__][INFO] - Starting iteration 241.
[2025-11-13 09:31:28,959][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:31:28,959][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:37,873][__main__][INFO] - Number of regex retries in iteration 241: 0
[2025-11-13 09:31:37,874][__main__][INFO] - agents played in iteration 241 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:31:38,374][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:38,410][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:38,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:38,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:38,478][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:38,479][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:39,244][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:31:39,543][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:31:39,870][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:31:40,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:31:40,525][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:31:40,850][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:31:41,175][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:31:41,500][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:31:41,826][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:31:42,151][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:31:42,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:31:42,803][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:31:43,128][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:31:43,453][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:31:43,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:31:44,105][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:31:44,431][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:31:44,755][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:31:45,087][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:31:45,412][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:31:45,737][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:31:46,062][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:31:46,389][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:31:46,717][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:47,046][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:47,371][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:31:47,697][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:31:48,022][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:31:48,349][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:31:48,680][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:31:49,001][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:31:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:31:49,649][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:31:50,439][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:31:51,188][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:51,189][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:51,191][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:52,208][__main__][INFO] - Iteration 242 took 23s (38.34% Gen, 57.28% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 54m 28s. Estimated total time: 19h 22m 32s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 45s, 500 more iterations: 3h 13m 45s.
[2025-11-13 09:31:52,210][__main__][INFO] - Starting iteration 242.
[2025-11-13 09:31:52,213][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:31:52,214][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:00,911][__main__][INFO] - Number of regex retries in iteration 242: 0
[2025-11-13 09:32:00,912][__main__][INFO] - agents played in iteration 242 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:32:01,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:01,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:01,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:01,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:01,468][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:01,469][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:02,223][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:32:02,619][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:32:02,921][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:32:03,247][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:32:03,587][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:32:03,912][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:32:04,236][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:32:04,563][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:32:04,900][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:32:05,225][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:32:05,549][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:32:05,874][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:32:06,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:32:06,530][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:32:06,856][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:32:07,180][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:32:07,510][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:32:07,836][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:32:08,161][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:32:08,488][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:32:08,814][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:32:09,139][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:32:09,465][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:32:09,790][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:32:10,116][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:32:10,442][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:10,768][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:11,096][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:11,427][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:11,758][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:12,084][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:12,734][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:13,475][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:14,236][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:14,237][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:14,239][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:15,247][__main__][INFO] - Iteration 243 took 23s (37.76% Gen, 57.86% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 43m 17s. Estimated total time: 19h 11m 44s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 57s.
[2025-11-13 09:32:15,250][__main__][INFO] - Starting iteration 243.
[2025-11-13 09:32:15,253][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:32:15,254][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:23,491][__main__][INFO] - Number of regex retries in iteration 243: 0
[2025-11-13 09:32:23,492][__main__][INFO] - agents played in iteration 243 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:32:23,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:24,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:24,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:24,078][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:24,079][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:24,079][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:24,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:32:25,086][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:32:25,413][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:32:25,740][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:32:26,065][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:32:26,390][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:32:26,716][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:32:27,043][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:32:27,370][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:32:27,700][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:32:28,024][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:32:28,356][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:32:28,684][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:32:29,012][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:32:29,338][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:32:29,663][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:32:29,988][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:32:30,315][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:32:30,640][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:32:30,966][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:32:31,293][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:32:31,619][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:32:31,944][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:32:32,269][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:32:32,599][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:32:32,921][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:33,247][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:33,573][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:33,898][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:34,224][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:34,550][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:34,877][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:35,203][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:35,943][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:36,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:36,705][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:36,707][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:37,700][__main__][INFO] - Iteration 244 took 22s (36.70% Gen, 58.87% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 13m 35s. Estimated total time: 18h 42m 24s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 24s, 500 more iterations: 3h 7m 4s.
[2025-11-13 09:32:37,703][__main__][INFO] - Starting iteration 244.
[2025-11-13 09:32:37,706][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:32:37,706][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:45,892][__main__][INFO] - Number of regex retries in iteration 244: 0
[2025-11-13 09:32:45,892][__main__][INFO] - agents played in iteration 244 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:32:46,356][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:46,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:46,423][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:46,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:46,456][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:46,457][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:47,156][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:32:47,452][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:32:47,782][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:32:48,112][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:32:48,438][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:32:48,761][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:32:49,086][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:32:49,417][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:32:49,743][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:32:50,067][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:32:50,395][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:32:50,721][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:32:51,050][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:32:51,376][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:32:51,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:32:52,032][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:32:52,365][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:32:52,701][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:32:53,030][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:32:53,356][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:32:53,682][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:32:54,016][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:32:54,341][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:32:54,665][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:32:54,990][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:32:55,320][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:55,970][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:56,296][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:56,624][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:56,951][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:57,276][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:57,602][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:58,351][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:59,080][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:59,081][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:59,083][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:00,048][__main__][INFO] - Iteration 245 took 22s (36.64% Gen, 59.04% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 7m 56s. Estimated total time: 18h 37m 7s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 14s, 500 more iterations: 3h 6m 11s.
[2025-11-13 09:33:00,050][__main__][INFO] - Starting iteration 245.
[2025-11-13 09:33:00,053][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:33:00,054][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:08,537][__main__][INFO] - Number of regex retries in iteration 245: 0
[2025-11-13 09:33:08,538][__main__][INFO] - agents played in iteration 245 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:33:08,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:09,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:09,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:09,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:09,097][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:09,097][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:09,811][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:10,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:10,432][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:10,756][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:11,079][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:11,404][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:11,730][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:12,054][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:12,389][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:12,716][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:13,043][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:13,367][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:13,699][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:14,023][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:14,350][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:14,675][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:15,015][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:33:15,348][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:33:15,677][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:33:16,008][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:33:16,336][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:33:16,667][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:33:16,991][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:33:17,316][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:33:17,640][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:33:17,965][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:33:18,291][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:33:18,617][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:33:18,944][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:33:19,269][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:33:19,595][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:33:19,920][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:33:20,249][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:33:20,994][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:33:21,731][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:33:21,734][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:33:21,735][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:22,676][__main__][INFO] - Iteration 246 took 22s (37.50% Gen, 58.33% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 21m 39s. Estimated total time: 18h 51m 13s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 42s, 500 more iterations: 3h 8m 32s.
[2025-11-13 09:33:22,678][__main__][INFO] - Starting iteration 246.
[2025-11-13 09:33:22,682][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:33:22,682][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:30,725][__main__][INFO] - Number of regex retries in iteration 246: 0
[2025-11-13 09:33:30,726][__main__][INFO] - agents played in iteration 246 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:33:31,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:31,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:31,254][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:31,287][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:31,287][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:31,288][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:31,991][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:32,286][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:32,612][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:32,934][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:33,259][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:33,585][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:33,908][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:34,233][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:34,557][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:34,883][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:35,212][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:35,538][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:35,863][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:36,189][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:36,513][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:36,839][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:37,163][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:33:37,492][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:33:37,817][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:33:38,147][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:33:38,473][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:33:38,801][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:33:39,126][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:33:39,454][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:33:39,782][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:33:40,108][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:33:40,435][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:33:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:33:41,086][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:33:41,412][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:33:41,738][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:33:42,063][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:33:42,389][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:33:43,131][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:33:43,857][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:33:43,860][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:33:43,864][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:44,837][__main__][INFO] - Iteration 247 took 22s (36.30% Gen, 59.30% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 57m 55s. Estimated total time: 18h 27m 51s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 55s, 500 more iterations: 3h 4m 38s.
[2025-11-13 09:33:44,839][__main__][INFO] - Starting iteration 247.
[2025-11-13 09:33:44,842][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:33:44,842][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:53,533][__main__][INFO] - Number of regex retries in iteration 247: 0
[2025-11-13 09:33:53,533][__main__][INFO] - agents played in iteration 247 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:33:53,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:54,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:54,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:54,095][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:54,095][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:54,096][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:54,797][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:55,093][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:55,421][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:55,746][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:56,071][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:56,397][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:56,722][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:57,047][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:57,373][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:57,698][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:58,024][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:58,351][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:58,678][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:59,004][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:59,331][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:59,655][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:59,979][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:34:00,305][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:34:00,630][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:34:00,954][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:34:01,282][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:34:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:34:01,933][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:34:02,258][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:34:02,584][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:34:02,912][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:34:03,239][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:34:03,564][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:03,894][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:04,225][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:04,551][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:04,877][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:05,202][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:05,945][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:06,686][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:06,687][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:06,689][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:07,604][__main__][INFO] - Iteration 248 took 22s (38.18% Gen, 57.79% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 27m 50s. Estimated total time: 18h 58m 9s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 56s, 500 more iterations: 3h 9m 41s.
[2025-11-13 09:34:07,606][__main__][INFO] - Starting iteration 248.
[2025-11-13 09:34:07,609][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:34:07,612][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:34:15,998][__main__][INFO] - Number of regex retries in iteration 248: 0
[2025-11-13 09:34:15,998][__main__][INFO] - agents played in iteration 248 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:34:16,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:16,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:16,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:16,561][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:16,561][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:34:16,562][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:34:17,272][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:34:17,569][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:34:17,897][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:34:18,226][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:34:18,553][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:34:18,882][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:34:19,207][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:34:19,532][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:34:19,856][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:34:20,182][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:34:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:34:20,831][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:34:21,162][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:34:21,490][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:34:21,816][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:34:22,143][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:34:22,466][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:34:22,791][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:34:23,115][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:34:23,439][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:34:23,763][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:34:24,088][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:34:24,411][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:34:24,740][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:34:25,064][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:34:25,388][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:34:25,715][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:34:26,039][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:26,365][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:26,690][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:27,021][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:27,348][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:27,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:28,438][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:29,165][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:29,167][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:29,168][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:30,067][__main__][INFO] - Iteration 249 took 22s (37.33% Gen, 58.64% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 12m 15s. Estimated total time: 18h 42m 57s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 25s, 500 more iterations: 3h 7m 9s.
[2025-11-13 09:34:30,070][__main__][INFO] - Starting iteration 249.
[2025-11-13 09:34:30,072][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:34:30,073][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:34:40,043][__main__][INFO] - Number of regex retries in iteration 249: 0
[2025-11-13 09:34:40,043][__main__][INFO] - agents played in iteration 249 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:34:40,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:40,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:40,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:40,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:40,620][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:34:40,621][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:34:41,330][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:34:41,624][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:34:41,954][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:34:42,280][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:34:42,605][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:34:42,932][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:34:43,254][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:34:43,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:34:43,903][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:34:44,233][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:34:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:34:44,880][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:34:45,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:34:45,531][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:34:45,856][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:34:46,182][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:34:46,508][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:34:46,833][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:34:47,158][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:34:47,482][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:34:47,807][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:34:48,131][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:34:48,456][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:34:48,781][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:34:49,105][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:34:49,429][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:34:49,753][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:34:50,077][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:50,403][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:50,729][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:51,055][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:51,382][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:51,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:52,484][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:53,228][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:53,229][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:53,231][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:54,267][__main__][INFO] - Iteration 250 took 24s (41.21% Gen, 54.51% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 38m 41s. Estimated total time: 20h 9m 46s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 19s, 500 more iterations: 3h 21m 37s.
[2025-11-13 09:34:54,269][__main__][INFO] - Starting iteration 250.
[2025-11-13 09:34:54,272][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:34:54,272][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:04,429][__main__][INFO] - Number of regex retries in iteration 250: 0
[2025-11-13 09:35:04,429][__main__][INFO] - agents played in iteration 250 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:35:04,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:04,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:04,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:05,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:05,003][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:05,004][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:05,736][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:06,031][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:06,358][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:06,684][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:07,011][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:07,339][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:07,669][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:07,997][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:08,321][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:08,647][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:08,971][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:09,296][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:09,620][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:09,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:10,273][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:10,600][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:10,923][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:11,248][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:35:11,572][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:35:11,897][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:35:12,223][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:35:12,551][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:35:12,877][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:35:13,201][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:35:13,524][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:35:13,851][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:35:14,175][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:35:14,499][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:35:14,824][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:35:15,151][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:35:15,479][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:35:15,806][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:35:16,144][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:35:16,907][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:35:17,679][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:17,681][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:17,682][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:35:19,490][__main__][INFO] - Iteration 251 took 25s (40.27% Gen, 52.55% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 29m 26s. Estimated total time: 21h 0m 56s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 1s, 500 more iterations: 3h 30m 9s.
[2025-11-13 09:35:19,492][__main__][INFO] - Starting iteration 251.
[2025-11-13 09:35:19,494][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:35:19,495][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:29,133][__main__][INFO] - Number of regex retries in iteration 251: 0
[2025-11-13 09:35:29,133][__main__][INFO] - agents played in iteration 251 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:35:29,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:29,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:29,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:29,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:29,729][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:29,730][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:30,486][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:30,782][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:31,107][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:31,433][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:31,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:32,083][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:32,409][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:32,733][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:33,059][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:33,383][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:33,706][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:34,030][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:34,354][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:34,679][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:35,003][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:35,328][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:35,652][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:35,976][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:35:36,301][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:35:36,624][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:35:36,948][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:35:37,272][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:35:37,596][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:35:37,919][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:35:38,243][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:35:38,568][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:35:38,891][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:35:39,214][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:35:39,539][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:35:39,868][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:35:40,190][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:35:40,515][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:35:40,843][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:35:41,617][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:35:42,327][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:42,329][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:42,331][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:35:43,296][__main__][INFO] - Iteration 252 took 23s (40.49% Gen, 55.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 18m 13s. Estimated total time: 19h 50m 7s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 40s, 500 more iterations: 3h 18m 21s.
[2025-11-13 09:35:43,298][__main__][INFO] - Starting iteration 252.
[2025-11-13 09:35:43,301][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:35:43,302][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:53,070][__main__][INFO] - Number of regex retries in iteration 252: 0
[2025-11-13 09:35:53,070][__main__][INFO] - agents played in iteration 252 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:35:53,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:53,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:53,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:53,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:53,656][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:53,656][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:54,361][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:54,657][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:54,984][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:55,311][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:55,638][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:55,969][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:56,296][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:56,621][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:56,957][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:57,278][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:57,604][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:57,928][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:58,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:58,577][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:58,902][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:59,227][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:59,550][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:59,874][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:36:00,197][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:36:00,525][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:00,853][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:01,181][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:01,505][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:01,831][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:02,157][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:02,483][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:02,808][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:03,133][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:03,458][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:03,783][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:04,109][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:04,435][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:04,760][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:05,529][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:06,268][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:06,269][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:06,271][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:07,201][__main__][INFO] - Iteration 253 took 23s (40.87% Gen, 55.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 22m 44s. Estimated total time: 19h 55m 3s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 10s.
[2025-11-13 09:36:07,203][__main__][INFO] - Starting iteration 253.
[2025-11-13 09:36:07,206][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:07,206][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:36:16,996][__main__][INFO] - Number of regex retries in iteration 253: 0
[2025-11-13 09:36:16,997][__main__][INFO] - agents played in iteration 253 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:36:17,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,586][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:36:17,586][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:36:18,345][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:36:18,642][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:36:18,966][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:36:19,292][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:36:19,622][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:36:19,948][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:36:20,275][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:36:20,602][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:36:20,933][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:36:21,260][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:36:21,585][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:36:21,911][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:36:22,239][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:36:22,564][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:36:22,895][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:36:23,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:36:23,552][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:36:23,880][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:36:24,206][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:36:24,534][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:24,860][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:25,512][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:25,841][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:26,168][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:26,492][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:26,816][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:27,140][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:27,466][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:27,789][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:28,116][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:28,446][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:28,778][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:29,531][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:30,300][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:30,302][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:30,303][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:31,320][__main__][INFO] - Iteration 254 took 24s (40.60% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 33m 3s. Estimated total time: 20h 5m 45s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 11s, 500 more iterations: 3h 20m 57s.
[2025-11-13 09:36:31,324][__main__][INFO] - Starting iteration 254.
[2025-11-13 09:36:31,327][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:31,328][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:36:42,164][__main__][INFO] - Number of regex retries in iteration 254: 0
[2025-11-13 09:36:42,165][__main__][INFO] - agents played in iteration 254 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:36:42,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:42,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:42,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:42,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:42,757][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:36:42,757][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:36:43,512][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:36:43,808][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:36:44,135][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:36:44,465][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:36:44,787][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:36:45,114][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:36:45,439][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:36:45,771][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:36:46,089][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:36:46,414][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:36:46,738][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:36:47,068][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:36:47,388][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:36:47,713][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:36:48,037][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:36:48,360][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:36:48,685][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:36:49,009][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:36:49,332][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:36:49,658][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:49,982][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:50,307][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:50,631][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:50,956][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:51,283][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:51,609][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:51,934][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:52,258][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:52,583][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:52,908][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:53,560][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:53,885][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:54,645][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:55,412][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:55,413][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:55,415][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:56,336][__main__][INFO] - Iteration 255 took 25s (43.33% Gen, 52.98% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 17m 22s. Estimated total time: 20h 50m 30s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 41s, 500 more iterations: 3h 28m 25s.
[2025-11-13 09:36:56,338][__main__][INFO] - Starting iteration 255.
[2025-11-13 09:36:56,341][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:56,342][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:06,303][__main__][INFO] - Number of regex retries in iteration 255: 0
[2025-11-13 09:37:06,304][__main__][INFO] - agents played in iteration 255 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:37:06,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,923][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:06,923][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:07,670][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:07,966][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:08,294][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:08,622][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:08,948][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:09,272][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:09,598][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:09,922][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:10,246][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:10,571][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:10,895][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:37:11,220][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:37:11,548][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:37:11,873][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:37:12,201][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:37:12,524][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:37:12,849][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:37:13,173][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:37:13,498][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:37:13,822][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:37:14,145][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:37:14,469][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:37:14,793][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:37:15,118][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:37:15,444][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:37:15,767][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:37:16,091][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:37:16,415][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:37:16,740][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:37:17,065][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:37:17,391][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:37:17,718][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:37:18,044][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:37:18,798][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:37:19,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:37:19,549][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:37:19,550][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:37:20,470][__main__][INFO] - Iteration 256 took 24s (41.28% Gen, 54.90% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 32m 57s. Estimated total time: 20h 6m 29s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 12s, 500 more iterations: 3h 21m 4s.
[2025-11-13 09:37:20,472][__main__][INFO] - Starting iteration 256.
[2025-11-13 09:37:20,475][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:37:20,475][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:30,542][__main__][INFO] - Number of regex retries in iteration 256: 0
[2025-11-13 09:37:30,542][__main__][INFO] - agents played in iteration 256 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:37:31,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:31,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:31,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:31,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:31,135][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:31,136][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:31,904][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:32,200][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:32,525][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:32,850][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:33,176][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:33,500][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:34,150][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:34,477][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:34,806][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:35,133][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:37:35,460][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:37:35,790][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:37:36,114][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:37:36,440][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:37:36,764][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:37:37,088][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:37:37,413][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:37:37,736][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:37:38,063][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:37:38,387][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:37:38,710][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:37:39,035][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:37:39,363][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:37:39,693][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:37:40,018][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:37:40,347][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:37:40,673][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:37:40,997][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:37:41,321][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:37:41,646][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:37:41,973][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:37:42,298][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:37:43,078][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:37:43,849][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:37:43,850][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:37:43,852][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:37:44,961][__main__][INFO] - Iteration 257 took 24s (41.11% Gen, 54.36% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 50m 24s. Estimated total time: 20h 24m 20s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 3s.
[2025-11-13 09:37:44,963][__main__][INFO] - Starting iteration 257.
[2025-11-13 09:37:44,965][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:37:44,966][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:55,214][__main__][INFO] - Number of regex retries in iteration 257: 0
[2025-11-13 09:37:55,214][__main__][INFO] - agents played in iteration 257 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:37:55,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,819][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:55,819][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:56,567][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:56,864][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:57,192][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:57,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:57,844][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:58,170][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:58,496][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:58,822][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:59,147][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:59,473][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:59,796][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:38:00,121][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:00,445][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:00,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:01,097][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:01,421][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:01,745][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:02,073][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:02,397][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:02,722][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:03,047][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:04,026][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:04,351][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:04,677][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:05,004][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:05,328][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:05,652][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:05,990][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:06,308][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:06,961][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:07,750][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:08,506][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:08,507][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:08,509][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:09,445][__main__][INFO] - Iteration 258 took 24s (41.86% Gen, 54.31% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 49m 39s. Estimated total time: 20h 24m 0s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 0s.
[2025-11-13 09:38:09,447][__main__][INFO] - Starting iteration 258.
[2025-11-13 09:38:09,449][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:38:09,450][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:38:19,874][__main__][INFO] - Number of regex retries in iteration 258: 0
[2025-11-13 09:38:19,875][__main__][INFO] - agents played in iteration 258 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:38:20,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,467][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:38:20,468][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:38:21,247][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:38:21,544][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:38:21,871][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:38:22,201][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:38:22,528][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:38:22,856][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:38:23,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:38:23,513][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:38:23,841][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:38:24,169][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:38:24,500][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:38:24,830][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:25,157][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:25,481][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:25,806][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:26,133][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:26,459][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:26,783][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:27,114][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:27,439][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:27,767][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:28,096][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:28,428][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:28,757][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:29,082][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:29,412][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:29,739][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:30,065][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:30,395][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:30,725][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:31,053][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:31,380][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:31,708][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:32,472][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:33,212][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:33,213][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:33,215][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:34,177][__main__][INFO] - Iteration 259 took 24s (42.16% Gen, 53.95% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 1m 41s. Estimated total time: 20h 36m 26s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 12s, 500 more iterations: 3h 26m 4s.
[2025-11-13 09:38:34,179][__main__][INFO] - Starting iteration 259.
[2025-11-13 09:38:34,182][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:38:34,182][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:38:44,403][__main__][INFO] - Number of regex retries in iteration 259: 0
[2025-11-13 09:38:44,403][__main__][INFO] - agents played in iteration 259 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:38:44,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,924][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,991][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,992][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:38:44,992][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:38:45,756][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:38:46,054][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:38:46,379][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:38:46,712][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:38:47,031][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:38:47,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:38:47,681][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:38:48,011][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:38:48,331][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:38:48,655][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:38:48,980][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:38:49,312][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:49,633][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:49,958][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:50,284][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:50,614][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:50,934][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:51,261][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:51,586][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:51,916][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:52,238][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:52,563][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:52,888][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:53,215][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:53,538][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:53,863][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:54,189][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:54,515][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:54,837][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:55,162][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:55,488][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:55,815][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:56,144][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:56,912][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:57,671][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:57,673][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:57,675][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:58,661][__main__][INFO] - Iteration 260 took 24s (41.75% Gen, 54.21% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 48m 51s. Estimated total time: 20h 24m 1s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 0s.
[2025-11-13 09:38:58,664][__main__][INFO] - Starting iteration 260.
[2025-11-13 09:38:58,666][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:38:58,667][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:08,947][__main__][INFO] - Number of regex retries in iteration 260: 0
[2025-11-13 09:39:08,948][__main__][INFO] - agents played in iteration 260 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:39:09,425][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:09,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:09,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:09,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:09,527][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:09,528][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:39:10,310][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:39:10,608][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:39:10,938][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:39:11,265][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:39:11,599][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:39:11,922][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:39:12,251][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:39:12,576][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:39:12,903][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:39:13,229][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:39:13,555][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:39:13,880][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:39:14,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:39:14,529][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:39:14,854][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:39:15,179][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:39:15,505][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:39:15,830][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:39:16,157][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:39:16,483][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:39:16,808][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:39:17,135][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:39:17,460][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:39:17,785][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:39:18,110][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:39:18,436][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:39:18,762][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:39:19,089][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:39:19,418][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:39:19,744][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:39:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:39:20,396][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:39:20,721][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:39:21,479][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:39:22,234][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:39:22,235][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:39:22,237][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:39:24,158][__main__][INFO] - Iteration 261 took 25s (40.33% Gen, 52.13% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 39m 1s. Estimated total time: 21h 14m 36s. Time estimates for 10 more iterations: 4m 14s, 100 more iterations: 42m 29s, 500 more iterations: 3h 32m 26s.
[2025-11-13 09:39:24,160][__main__][INFO] - Starting iteration 261.
[2025-11-13 09:39:24,162][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:39:24,163][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:34,280][__main__][INFO] - Number of regex retries in iteration 261: 0
[2025-11-13 09:39:34,280][__main__][INFO] - agents played in iteration 261 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:39:34,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:34,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:34,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,242][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:35,242][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:39:35,994][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:39:36,291][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:39:36,625][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:39:36,944][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:39:37,269][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:39:37,596][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:39:37,926][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:39:38,247][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:39:38,574][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:39:38,900][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:39:39,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:39:39,552][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:39:39,875][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:39:40,200][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:39:40,530][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:39:40,853][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:39:41,183][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:39:41,510][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:39:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:39:42,164][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:39:42,490][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:39:42,815][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:39:43,140][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:39:43,464][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:39:43,788][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:39:44,113][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:39:44,438][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:39:44,764][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:39:45,088][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:39:45,412][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:39:45,738][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:39:46,063][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:39:46,388][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:39:47,166][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:39:47,911][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:39:47,913][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:39:47,914][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:39:48,959][__main__][INFO] - Iteration 262 took 24s (40.80% Gen, 54.98% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 3m 53s. Estimated total time: 20h 39m 53s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 19s, 500 more iterations: 3h 26m 38s.
[2025-11-13 09:39:48,962][__main__][INFO] - Starting iteration 262.
[2025-11-13 09:39:48,965][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:39:48,966][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:59,376][__main__][INFO] - Number of regex retries in iteration 262: 0
[2025-11-13 09:39:59,377][__main__][INFO] - agents played in iteration 262 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:39:59,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,967][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:59,968][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:00,754][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:01,050][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:01,376][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:02,027][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:02,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:02,682][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:03,010][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:03,338][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:03,662][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:03,987][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:04,317][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:04,644][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:04,968][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:05,293][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:05,951][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:06,280][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:06,608][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:06,939][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:07,267][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:07,592][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:07,918][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:08,244][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:08,570][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:08,894][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:09,221][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:09,548][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:09,877][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:10,205][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:40:10,529][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:40:10,855][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:40:11,180][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:40:11,964][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:40:12,726][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:40:12,727][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:40:12,730][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:40:13,693][__main__][INFO] - Iteration 263 took 24s (42.10% Gen, 54.00% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 0m 3s. Estimated total time: 20h 36m 28s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 12s, 500 more iterations: 3h 26m 4s.
[2025-11-13 09:40:13,695][__main__][INFO] - Starting iteration 263.
[2025-11-13 09:40:13,698][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:40:13,699][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:40:24,634][__main__][INFO] - Number of regex retries in iteration 263: 0
[2025-11-13 09:40:24,635][__main__][INFO] - agents played in iteration 263 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:40:25,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:25,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:25,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:25,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:25,250][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:40:25,251][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:25,991][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:26,288][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:26,616][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:26,941][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:27,266][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:27,591][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:27,921][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:28,248][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:28,897][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:29,224][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:29,550][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:29,874][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:30,202][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:30,532][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:30,856][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:31,186][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:31,513][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:31,841][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:32,166][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:32,490][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:32,817][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:33,144][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:33,468][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:33,792][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:34,119][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:34,449][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:34,775][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:35,100][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:35,425][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:40:35,752][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:40:36,084][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:40:36,411][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:40:37,214][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:40:37,965][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:40:37,966][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:40:37,968][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:40:38,897][__main__][INFO] - Iteration 264 took 25s (43.40% Gen, 52.91% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 23m 7s. Estimated total time: 20h 59m 57s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 59s, 500 more iterations: 3h 29m 59s.
[2025-11-13 09:40:38,899][__main__][INFO] - Starting iteration 264.
[2025-11-13 09:40:38,902][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:40:38,902][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:40:48,748][__main__][INFO] - Number of regex retries in iteration 264: 0
[2025-11-13 09:40:48,749][__main__][INFO] - agents played in iteration 264 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:40:49,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,349][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:40:49,349][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:50,128][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:50,424][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:50,751][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:51,077][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:51,403][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:51,729][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:52,054][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:52,379][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:52,704][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:53,029][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:53,353][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:53,677][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:54,003][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:54,329][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:54,655][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:54,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:55,311][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:55,630][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:55,955][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:56,281][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:56,606][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:56,933][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:57,257][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:57,582][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:57,917][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:58,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:58,558][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:58,888][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:59,224][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:59,550][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:40:59,878][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:00,203][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:00,528][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:01,276][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:02,031][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:02,033][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:02,035][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:03,031][__main__][INFO] - Iteration 265 took 24s (40.81% Gen, 55.06% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 29m 16s. Estimated total time: 20h 6m 30s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 13s, 500 more iterations: 3h 21m 5s.
[2025-11-13 09:41:03,033][__main__][INFO] - Starting iteration 265.
[2025-11-13 09:41:03,037][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:03,037][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:41:13,401][__main__][INFO] - Number of regex retries in iteration 265: 0
[2025-11-13 09:41:13,402][__main__][INFO] - agents played in iteration 265 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:41:13,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:13,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:13,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,000][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,001][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:41:14,001][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:41:14,774][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:41:15,071][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:41:15,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:41:15,721][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:41:16,046][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:41:16,371][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:41:16,695][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:41:17,020][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:41:17,344][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:41:17,669][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:41:17,994][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:41:18,319][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:41:18,644][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:41:18,968][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:41:19,294][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:41:19,621][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:41:19,944][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:41:20,269][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:41:20,594][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:41:20,918][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:41:21,243][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:41:21,568][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:41:21,893][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:41:22,218][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:41:22,543][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:41:22,868][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:41:23,195][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:41:23,520][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:41:23,846][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:41:24,170][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:41:24,504][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:24,828][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:25,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:25,867][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:26,607][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:26,609][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:26,610][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:27,620][__main__][INFO] - Iteration 266 took 24s (42.16% Gen, 53.73% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 51m 34s. Estimated total time: 20h 29m 13s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 58s, 500 more iterations: 3h 24m 52s.
[2025-11-13 09:41:27,622][__main__][INFO] - Starting iteration 266.
[2025-11-13 09:41:27,625][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:27,626][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:41:37,068][mllm.models.large_language_model_local][WARNING] - Response <B>
user
Last round, the other agent played <A>. did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 09:41:38,560][__main__][INFO] - Number of regex retries in iteration 266: 1
[2025-11-13 09:41:38,560][__main__][INFO] - agents played in iteration 266 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:41:39,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,111][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,146][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:41:39,146][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:41:39,922][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:41:40,221][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:41:40,546][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:41:40,872][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:41:41,197][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:41:41,533][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:41:41,858][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:41:42,183][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:41:42,507][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:41:42,840][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:41:43,165][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:41:43,491][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:41:43,816][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:41:44,143][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:41:44,469][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:41:44,795][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:41:45,120][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:41:45,446][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:41:45,771][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:41:46,096][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:41:46,421][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:41:46,747][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:41:47,072][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:41:47,396][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:41:47,720][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:41:48,045][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:41:48,370][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:41:48,694][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:41:49,019][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:41:49,345][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:41:49,669][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:49,994][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:50,320][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:51,041][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:51,805][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:51,806][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:51,808][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:52,796][__main__][INFO] - Iteration 267 took 25s (43.44% Gen, 52.63% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 20m 30s. Estimated total time: 20h 58m 34s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 57s, 500 more iterations: 3h 29m 45s.
[2025-11-13 09:41:52,798][__main__][INFO] - Starting iteration 267.
[2025-11-13 09:41:52,802][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:52,802][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:03,420][__main__][INFO] - Number of regex retries in iteration 267: 0
[2025-11-13 09:42:03,421][__main__][INFO] - agents played in iteration 267 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:42:03,914][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:03,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:03,981][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:04,014][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:04,014][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:04,015][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:04,789][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:05,086][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:05,414][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:05,741][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:06,066][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:06,391][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:06,717][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:07,041][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:07,366][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:07,690][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:08,014][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:08,341][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:08,669][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:08,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:09,319][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:09,643][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:09,969][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:10,292][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:42:10,618][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:42:10,947][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:42:11,278][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:42:11,606][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:42:11,930][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:42:12,256][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:42:12,581][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:42:12,910][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:42:13,235][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:42:13,561][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:42:13,888][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:42:14,211][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:42:14,537][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:42:14,862][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:42:15,187][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:42:15,940][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:42:16,714][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:42:16,715][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:42:16,717][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:42:17,699][__main__][INFO] - Iteration 268 took 24s (42.65% Gen, 53.40% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 6m 25s. Estimated total time: 20h 44m 54s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 29s, 500 more iterations: 3h 27m 29s.
[2025-11-13 09:42:17,701][__main__][INFO] - Starting iteration 268.
[2025-11-13 09:42:17,704][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:42:17,705][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:28,329][__main__][INFO] - Number of regex retries in iteration 268: 0
[2025-11-13 09:42:28,330][__main__][INFO] - agents played in iteration 268 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:42:28,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,921][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:28,922][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:29,687][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:30,310][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:30,635][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:30,963][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:31,610][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:31,935][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:32,265][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:32,587][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:32,911][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:33,235][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:33,565][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:33,884][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:34,210][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:34,534][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:34,863][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:35,184][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:42:35,509][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:42:35,834][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:42:36,166][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:42:36,487][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:42:36,816][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:42:37,143][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:42:37,468][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:42:37,801][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:42:38,129][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:42:38,455][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:42:38,780][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:42:39,114][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:42:39,439][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:42:39,764][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:42:40,090][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:42:40,855][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:42:41,606][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:42:41,608][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:42:41,609][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:42:42,592][__main__][INFO] - Iteration 269 took 24s (42.69% Gen, 53.36% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 5m 32s. Estimated total time: 20h 44m 25s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 28s, 500 more iterations: 3h 27m 24s.
[2025-11-13 09:42:42,596][__main__][INFO] - Starting iteration 269.
[2025-11-13 09:42:42,598][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:42:42,599][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:53,099][__main__][INFO] - Number of regex retries in iteration 269: 0
[2025-11-13 09:42:53,099][__main__][INFO] - agents played in iteration 269 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:42:53,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:53,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:53,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:53,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:53,707][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:53,707][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:54,444][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:54,741][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:55,066][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:55,390][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:55,714][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:56,038][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:56,363][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:56,687][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:57,011][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:57,337][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:57,663][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:57,989][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:58,314][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:58,639][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:58,963][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:59,288][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:59,613][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:59,938][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:43:00,265][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:43:00,592][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:00,917][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:01,241][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:01,567][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:01,894][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:02,220][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:02,546][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:02,872][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:03,200][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:03,525][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:03,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:04,178][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:04,503][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:04,830][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:05,584][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:06,323][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:06,324][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:06,326][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:07,351][__main__][INFO] - Iteration 270 took 24s (42.42% Gen, 53.43% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 58m 23s. Estimated total time: 20h 37m 41s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 15s, 500 more iterations: 3h 26m 16s.
[2025-11-13 09:43:07,353][__main__][INFO] - Starting iteration 270.
[2025-11-13 09:43:07,356][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:43:07,357][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:43:18,220][__main__][INFO] - Number of regex retries in iteration 270: 0
[2025-11-13 09:43:18,221][__main__][INFO] - agents played in iteration 270 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:43:18,717][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:18,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:18,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:18,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:18,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:43:18,839][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:43:19,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:43:19,916][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:43:20,244][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:43:20,570][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:43:20,897][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:43:21,224][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:43:21,553][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:43:21,880][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:43:22,207][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:43:22,533][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:43:22,861][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:43:23,187][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:43:23,512][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:43:23,836][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:43:24,163][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:43:24,487][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:43:24,812][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:43:25,136][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:43:25,463][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:43:25,789][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:26,114][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:26,441][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:26,773][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:27,098][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:27,425][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:27,756][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:28,083][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:28,411][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:28,738][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:29,069][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:29,396][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:29,724][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:30,050][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:30,835][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:31,584][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:31,585][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:31,587][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:33,470][__main__][INFO] - Iteration 271 took 26s (41.60% Gen, 51.18% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 5m 59s. Estimated total time: 21h 45m 43s. Time estimates for 10 more iterations: 4m 21s, 100 more iterations: 43m 31s, 500 more iterations: 3h 37m 37s.
[2025-11-13 09:43:33,473][__main__][INFO] - Starting iteration 271.
[2025-11-13 09:43:33,475][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:43:33,476][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:43:44,112][__main__][INFO] - Number of regex retries in iteration 271: 0
[2025-11-13 09:43:44,113][__main__][INFO] - agents played in iteration 271 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:43:44,602][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,705][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:43:44,705][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:43:45,407][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:43:45,702][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:43:46,028][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:43:46,351][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:43:46,676][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:43:47,001][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:43:47,327][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:43:47,653][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:43:47,977][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:43:48,302][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:43:48,627][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:43:48,952][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:43:49,277][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:43:49,601][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:43:49,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:43:50,253][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:43:50,577][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:43:50,904][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:43:51,229][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:43:51,554][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:51,878][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:52,203][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:52,530][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:52,859][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:53,186][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:53,508][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:53,834][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:54,158][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:54,485][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:54,807][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:55,133][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:55,458][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:55,783][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:56,540][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:57,305][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:57,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:57,308][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:58,242][__main__][INFO] - Iteration 272 took 24s (42.95% Gen, 53.27% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 58m 12s. Estimated total time: 20h 38m 22s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 16s, 500 more iterations: 3h 26m 23s.
[2025-11-13 09:43:58,244][__main__][INFO] - Starting iteration 272.
[2025-11-13 09:43:58,247][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:43:58,247][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:08,812][__main__][INFO] - Number of regex retries in iteration 272: 0
[2025-11-13 09:44:08,813][__main__][INFO] - agents played in iteration 272 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:44:09,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:09,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:09,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:09,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:09,408][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:09,408][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:10,173][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:44:10,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:44:10,794][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:44:11,118][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:44:11,443][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:44:11,769][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:44:12,093][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:44:12,419][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:44:12,744][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:44:13,075][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:44:13,402][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:44:13,729][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:44:14,055][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:44:14,380][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:44:14,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:44:15,029][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:44:15,353][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:44:15,678][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:44:16,003][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:44:16,328][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:44:16,655][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:44:16,978][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:44:17,303][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:44:17,628][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:44:17,953][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:44:18,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:44:18,605][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:44:18,930][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:44:19,258][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:44:19,583][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:44:19,909][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:44:20,240][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:44:20,568][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:44:21,340][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:44:22,077][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:44:22,079][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:44:22,081][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:44:23,066][__main__][INFO] - Iteration 273 took 24s (42.57% Gen, 53.46% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 0m 27s. Estimated total time: 20h 41m 1s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 22s, 500 more iterations: 3h 26m 50s.
[2025-11-13 09:44:23,068][__main__][INFO] - Starting iteration 273.
[2025-11-13 09:44:23,071][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:44:23,072][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:33,639][__main__][INFO] - Number of regex retries in iteration 273: 0
[2025-11-13 09:44:33,639][__main__][INFO] - agents played in iteration 273 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:44:34,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:34,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:34,225][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:34,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:34,259][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:34,260][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:35,015][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:44:35,311][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:44:35,638][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:44:35,963][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:44:36,290][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:44:36,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:44:36,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:44:37,275][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:44:37,599][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:44:37,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:44:38,252][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:44:38,579][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:44:38,909][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:44:39,234][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:44:39,561][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:44:39,885][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:44:40,210][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:44:40,534][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:44:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:44:41,183][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:44:41,509][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:44:41,835][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:44:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:44:42,494][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:44:42,819][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:44:43,144][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:44:43,472][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:44:43,797][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:44:44,123][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:44:44,450][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:44:44,776][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:44:45,103][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:44:45,429][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:44:46,188][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:44:46,931][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:44:46,933][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:44:46,934][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:44:47,910][__main__][INFO] - Iteration 274 took 24s (42.54% Gen, 53.52% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 1m 1s. Estimated total time: 20h 42m 0s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 24s, 500 more iterations: 3h 27m 0s.
[2025-11-13 09:44:47,912][__main__][INFO] - Starting iteration 274.
[2025-11-13 09:44:47,916][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:44:47,916][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:58,796][__main__][INFO] - Number of regex retries in iteration 274: 0
[2025-11-13 09:44:58,796][__main__][INFO] - agents played in iteration 274 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:44:59,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:59,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:59,362][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:59,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:59,397][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:59,397][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:00,129][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:45:00,424][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:00,751][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:01,076][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:01,407][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:01,733][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:02,057][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:02,387][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:02,715][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:03,040][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:03,364][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:03,690][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:04,013][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:04,336][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:04,662][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:05,316][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:05,643][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:05,968][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:06,295][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:06,622][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:06,948][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:07,276][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:07,601][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:07,927][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:08,253][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:08,578][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:08,903][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:09,228][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:09,553][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:09,879][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:10,205][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:45:10,532][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:45:11,294][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:45:12,037][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:45:12,039][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:45:12,040][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:45:13,006][__main__][INFO] - Iteration 275 took 25s (43.36% Gen, 52.78% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 13m 10s. Estimated total time: 20h 54m 34s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 49s, 500 more iterations: 3h 29m 5s.
[2025-11-13 09:45:13,008][__main__][INFO] - Starting iteration 275.
[2025-11-13 09:45:13,011][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:45:13,012][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:45:23,131][__main__][INFO] - Number of regex retries in iteration 275: 0
[2025-11-13 09:45:23,132][__main__][INFO] - agents played in iteration 275 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:45:23,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,726][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:45:23,727][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:24,482][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:45:24,778][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:25,107][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:25,436][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:25,764][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:26,094][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:26,421][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:26,748][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:27,075][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:27,400][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:27,725][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:28,056][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:28,385][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:28,709][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:29,034][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:29,359][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:29,685][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:30,010][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:30,335][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:30,661][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:30,988][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:31,319][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:31,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:31,977][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:32,628][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:32,955][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:33,285][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:33,943][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:34,269][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:34,594][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:45:34,922][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:45:35,676][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:45:36,427][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:45:36,428][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:45:36,430][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:45:37,426][__main__][INFO] - Iteration 276 took 24s (41.45% Gen, 54.47% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 38m 57s. Estimated total time: 20h 20m 46s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 41s, 500 more iterations: 3h 23m 27s.
[2025-11-13 09:45:37,428][__main__][INFO] - Starting iteration 276.
[2025-11-13 09:45:37,432][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:45:37,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:45:48,566][__main__][INFO] - Number of regex retries in iteration 276: 0
[2025-11-13 09:45:48,567][__main__][INFO] - agents played in iteration 276 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:45:49,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:49,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:49,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:49,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:49,165][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:45:49,165][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:49,910][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:45:50,205][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:50,530][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:50,855][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:51,179][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:51,505][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:51,830][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:52,154][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:52,479][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:52,804][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:53,130][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:53,454][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:53,778][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:54,108][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:54,429][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:54,754][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:55,079][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:55,409][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:55,730][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:56,055][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:56,378][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:56,709][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:57,029][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:57,354][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:57,678][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:58,003][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:58,327][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:58,652][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:58,979][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:59,305][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:59,634][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:59,962][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:00,289][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:01,031][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:01,778][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:01,779][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:01,781][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:02,698][__main__][INFO] - Iteration 277 took 25s (44.07% Gen, 52.30% Train). Generation: 11s, Training: 13s. Estimated remaining time: 19h 21m 7s. Estimated total time: 21h 3m 21s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 6s, 500 more iterations: 3h 30m 33s.
[2025-11-13 09:46:02,703][__main__][INFO] - Starting iteration 277.
[2025-11-13 09:46:02,705][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:02,706][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:46:13,654][__main__][INFO] - Number of regex retries in iteration 277: 0
[2025-11-13 09:46:13,654][__main__][INFO] - agents played in iteration 277 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:46:14,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,209][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,244][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:46:14,244][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:46:15,007][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:46:15,303][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:46:15,629][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:46:15,954][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:46:16,278][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:46:16,603][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:46:16,928][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:46:17,253][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:46:17,578][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:46:17,903][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:46:18,228][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:46:18,553][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:46:18,876][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:46:19,201][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:46:19,526][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:46:19,852][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:46:20,177][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:46:20,505][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:46:20,830][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:46:21,157][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:46:21,488][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:46:21,808][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:46:22,134][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:46:22,459][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:46:22,784][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:46:23,111][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:46:23,435][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:46:23,759][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:46:24,085][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:46:24,408][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:46:24,732][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:46:25,057][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:25,387][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:26,170][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:26,930][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:26,932][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:26,933][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:27,887][__main__][INFO] - Iteration 278 took 25s (43.48% Gen, 52.73% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 16m 29s. Estimated total time: 20h 59m 8s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 58s, 500 more iterations: 3h 29m 51s.
[2025-11-13 09:46:27,889][__main__][INFO] - Starting iteration 278.
[2025-11-13 09:46:27,892][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:27,892][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:46:38,585][__main__][INFO] - Number of regex retries in iteration 278: 0
[2025-11-13 09:46:38,585][__main__][INFO] - agents played in iteration 278 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:46:39,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,189][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:46:39,190][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:46:39,970][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:46:40,266][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:46:40,591][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:46:40,916][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:46:41,241][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:46:41,566][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:46:41,895][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:46:42,222][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:46:42,552][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:46:42,878][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:46:43,204][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:46:43,532][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:46:43,857][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:46:44,182][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:46:44,507][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:46:44,833][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:46:45,157][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:46:45,492][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:46:45,817][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:46:46,142][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:46:46,466][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:46:46,793][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:46:47,117][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:46:47,441][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:46:47,766][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:46:48,091][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:46:48,417][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:46:48,740][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:46:49,064][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:46:49,389][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:46:49,715][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:46:50,039][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:50,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:51,146][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:51,921][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:51,923][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:51,924][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:52,860][__main__][INFO] - Iteration 279 took 24s (42.82% Gen, 53.42% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 5m 24s. Estimated total time: 20h 48m 28s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 36s, 500 more iterations: 3h 28m 4s.
[2025-11-13 09:46:52,863][__main__][INFO] - Starting iteration 279.
[2025-11-13 09:46:52,866][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:52,867][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:03,208][__main__][INFO] - Number of regex retries in iteration 279: 0
[2025-11-13 09:47:03,209][__main__][INFO] - agents played in iteration 279 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:47:03,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:03,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:03,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:03,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:03,819][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:03,819][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:04,577][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:04,874][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:05,210][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:05,537][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:05,865][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:06,193][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:06,518][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:06,845][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:07,170][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:07,497][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:07,823][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:08,150][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:08,480][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:47:08,810][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:47:09,136][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:47:09,460][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:47:09,790][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:47:10,115][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:47:10,440][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:47:10,767][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:47:11,093][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:47:11,424][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:47:11,749][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:47:12,075][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:47:12,402][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:47:12,727][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:47:13,051][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:47:13,374][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:47:13,699][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:47:14,031][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:47:14,350][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:47:14,678][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:47:15,003][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:47:15,774][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:47:16,519][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:47:16,520][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:47:16,522][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:47:17,426][__main__][INFO] - Iteration 280 took 24s (42.11% Gen, 54.21% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 44m 34s. Estimated total time: 20h 28m 2s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 56s, 500 more iterations: 3h 24m 40s.
[2025-11-13 09:47:17,428][__main__][INFO] - Starting iteration 280.
[2025-11-13 09:47:17,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:47:17,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:27,827][__main__][INFO] - Number of regex retries in iteration 280: 0
[2025-11-13 09:47:27,827][__main__][INFO] - agents played in iteration 280 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:47:28,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:28,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:28,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:28,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:28,415][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:28,415][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:29,166][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:29,462][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:29,788][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:30,115][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:30,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:30,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:31,098][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:31,426][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:31,751][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:32,078][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:32,402][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:32,728][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:33,051][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:47:33,376][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:47:33,702][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:47:34,031][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:47:34,350][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:47:34,677][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:47:35,002][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:47:35,338][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:47:35,659][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:47:35,984][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:47:36,309][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:47:36,639][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:47:36,958][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:47:37,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:47:37,607][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:47:37,940][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:47:38,258][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:47:38,586][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:47:38,910][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:47:39,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:47:39,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:47:40,319][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:47:41,051][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:47:41,053][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:47:41,054][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:47:42,954][__main__][INFO] - Iteration 281 took 25s (40.73% Gen, 51.82% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 32m 16s. Estimated total time: 21h 16m 10s. Time estimates for 10 more iterations: 4m 15s, 100 more iterations: 42m 32s, 500 more iterations: 3h 32m 41s.
[2025-11-13 09:47:42,956][__main__][INFO] - Starting iteration 281.
[2025-11-13 09:47:42,959][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:47:42,959][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:53,678][__main__][INFO] - Number of regex retries in iteration 281: 0
[2025-11-13 09:47:53,679][__main__][INFO] - agents played in iteration 281 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:47:54,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:54,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:54,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:54,282][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:54,283][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:54,283][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:55,031][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:55,327][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:55,652][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:55,979][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:56,307][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:56,632][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:56,956][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:57,280][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:57,605][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:57,930][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:58,256][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:58,579][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:58,903][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:47:59,229][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:47:59,553][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:47:59,878][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:00,202][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:00,530][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:00,855][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:01,181][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:01,505][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:01,832][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:02,158][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:02,490][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:02,816][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:03,141][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:03,468][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:03,794][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:04,118][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:04,444][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:04,769][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:05,095][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:05,422][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:06,195][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:06,924][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:06,925][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:06,927][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:07,846][__main__][INFO] - Iteration 282 took 24s (43.07% Gen, 53.23% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 0m 7s. Estimated total time: 20h 44m 26s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 28s, 500 more iterations: 3h 27m 24s.
[2025-11-13 09:48:07,849][__main__][INFO] - Starting iteration 282.
[2025-11-13 09:48:07,852][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:07,852][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:48:15,419][mllm.models.large_language_model_local][WARNING] - Response ABCDEFGHIJKLMNOPQRSTUVWXY<B> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 09:48:19,159][__main__][INFO] - Number of regex retries in iteration 282: 1
[2025-11-13 09:48:19,160][__main__][INFO] - agents played in iteration 282 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:48:19,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,757][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:48:19,758][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:48:20,479][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:48:20,774][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:48:21,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:48:21,427][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:48:21,755][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:48:22,079][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:48:22,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:48:22,733][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:48:23,061][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:48:23,385][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:48:23,712][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:48:24,037][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:48:24,365][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:48:24,692][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:48:25,016][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:48:25,340][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:25,664][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:25,991][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:26,316][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:26,642][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:26,967][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:27,291][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:27,618][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:27,943][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:28,268][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:28,594][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:28,927][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:29,254][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:29,578][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:29,904][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:30,229][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:30,555][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:30,880][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:31,657][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:32,374][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:32,375][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:32,377][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:33,293][__main__][INFO] - Iteration 283 took 25s (44.44% Gen, 51.95% Train). Generation: 11s, Training: 13s. Estimated remaining time: 19h 27m 21s. Estimated total time: 21h 12m 5s. Time estimates for 10 more iterations: 4m 14s, 100 more iterations: 42m 24s, 500 more iterations: 3h 32m 0s.
[2025-11-13 09:48:33,295][__main__][INFO] - Starting iteration 283.
[2025-11-13 09:48:33,298][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:33,298][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:48:44,338][__main__][INFO] - Number of regex retries in iteration 283: 0
[2025-11-13 09:48:44,339][__main__][INFO] - agents played in iteration 283 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:48:44,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:44,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:44,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:44,910][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:44,910][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:48:44,911][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:48:45,613][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:48:45,909][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:48:46,238][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:48:46,568][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:48:46,893][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:48:47,217][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:48:47,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:48:47,873][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:48:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:48:48,520][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:48:48,846][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:48:49,175][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:48:49,508][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:48:49,833][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:48:50,158][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:48:50,484][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:50,810][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:51,137][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:51,463][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:51,788][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:52,113][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:52,437][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:52,762][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:53,087][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:53,415][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:53,743][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:54,069][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:54,393][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:54,718][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:55,043][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:55,698][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:56,025][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:56,786][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:57,502][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:57,504][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:57,506][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:58,432][__main__][INFO] - Iteration 284 took 25s (43.92% Gen, 52.39% Train). Generation: 11s, Training: 13s. Estimated remaining time: 19h 11m 37s. Estimated total time: 20h 56m 46s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 53s, 500 more iterations: 3h 29m 27s.
[2025-11-13 09:48:58,435][__main__][INFO] - Starting iteration 284.
[2025-11-13 09:48:58,437][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:58,437][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:09,180][__main__][INFO] - Number of regex retries in iteration 284: 0
[2025-11-13 09:49:09,181][__main__][INFO] - agents played in iteration 284 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:49:09,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:09,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:09,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:09,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:09,767][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:09,768][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:10,484][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:49:10,780][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:49:11,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:49:11,434][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:49:11,760][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:49:12,085][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:49:12,410][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:49:12,734][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:49:13,063][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:49:13,389][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:49:13,714][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:49:14,040][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:49:14,369][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:49:14,694][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:49:15,018][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:49:15,346][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:49:15,668][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:49:15,997][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:49:16,324][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:49:16,657][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:49:16,977][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:49:17,304][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:49:17,629][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:49:17,955][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:49:18,285][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:49:18,616][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:49:18,941][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:49:19,269][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:49:19,597][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:49:19,922][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:49:20,248][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:49:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:49:20,901][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:49:21,679][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:49:22,405][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:49:22,407][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:49:22,408][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:49:23,294][__main__][INFO] - Iteration 285 took 24s (43.22% Gen, 53.21% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 57m 19s. Estimated total time: 20h 42m 54s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 25s, 500 more iterations: 3h 27m 9s.
[2025-11-13 09:49:23,296][__main__][INFO] - Starting iteration 285.
[2025-11-13 09:49:23,299][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:49:23,299][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:33,404][__main__][INFO] - Number of regex retries in iteration 285: 0
[2025-11-13 09:49:33,404][__main__][INFO] - agents played in iteration 285 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:49:33,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,998][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:33,999][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:34,737][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:49:35,033][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:49:35,365][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:49:35,687][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:49:36,011][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:49:36,337][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:49:36,661][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:49:36,983][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:49:37,308][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:49:37,633][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:49:37,957][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:49:38,281][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:49:38,605][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:49:38,931][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:49:39,262][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:49:39,582][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:49:39,906][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:49:40,231][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:49:40,562][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:49:40,881][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:49:41,206][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:49:41,530][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:49:41,860][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:49:42,180][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:49:42,505][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:49:42,830][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:49:43,158][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:49:43,480][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:49:43,804][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:49:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:49:44,463][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:49:44,788][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:49:45,118][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:49:45,915][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:49:46,826][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:49:46,827][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:49:46,829][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:49:47,832][__main__][INFO] - Iteration 286 took 24s (41.19% Gen, 54.72% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 40m 42s. Estimated total time: 20h 26m 41s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 53s, 500 more iterations: 3h 24m 26s.
[2025-11-13 09:49:47,834][__main__][INFO] - Starting iteration 286.
[2025-11-13 09:49:47,836][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:49:47,837][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:58,705][__main__][INFO] - Number of regex retries in iteration 286: 0
[2025-11-13 09:49:58,706][__main__][INFO] - agents played in iteration 286 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:49:59,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:59,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:59,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:59,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:59,324][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:59,324][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:50:00,068][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:50:00,365][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:00,693][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:01,016][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:01,342][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:01,666][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:01,997][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:02,322][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:02,648][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:02,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:03,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:03,629][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:03,955][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:04,284][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:04,609][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:04,934][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:05,262][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:05,587][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:05,911][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:06,236][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:06,564][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:06,893][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:07,219][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:07,546][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:07,871][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:08,200][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:08,528][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:08,857][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:09,188][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:09,518][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:09,849][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:10,177][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:50:10,505][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:50:11,289][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:50:12,009][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:50:12,011][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:50:12,013][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:50:13,020][__main__][INFO] - Iteration 287 took 25s (43.16% Gen, 52.84% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 12m 50s. Estimated total time: 20h 59m 15s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 58s, 500 more iterations: 3h 29m 52s.
[2025-11-13 09:50:13,022][__main__][INFO] - Starting iteration 287.
[2025-11-13 09:50:13,025][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:50:13,025][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:50:23,121][__main__][INFO] - Number of regex retries in iteration 287: 0
[2025-11-13 09:50:23,122][__main__][INFO] - agents played in iteration 287 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:50:23,600][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,701][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:50:23,702][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:50:24,475][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:50:24,773][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:25,097][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:25,423][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:25,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:26,081][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:26,405][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:26,730][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:27,066][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:27,391][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:27,716][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:28,042][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:28,378][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:28,702][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:29,026][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:29,353][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:29,685][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:30,012][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:30,343][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:30,672][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:30,995][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:31,325][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:31,656][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:31,989][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:32,308][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:32,635][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:32,962][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:33,296][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:33,615][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:33,940][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:34,265][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:34,597][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:50:34,917][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:50:35,670][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:50:36,396][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:50:36,398][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:50:36,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:50:37,382][__main__][INFO] - Iteration 288 took 24s (41.45% Gen, 54.52% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 31m 6s. Estimated total time: 20h 17m 55s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 35s, 500 more iterations: 3h 22m 59s.
[2025-11-13 09:50:37,385][__main__][INFO] - Starting iteration 288.
[2025-11-13 09:50:37,388][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:50:37,388][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:50:47,784][__main__][INFO] - Number of regex retries in iteration 288: 0
[2025-11-13 09:50:47,785][__main__][INFO] - agents played in iteration 288 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:50:48,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:50:48,370][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:50:49,130][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:50:49,428][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:49,753][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:50,079][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:50,407][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:50,736][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:51,062][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:51,393][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:51,724][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:52,050][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:52,376][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:52,704][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:53,028][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:53,354][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:53,684][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:54,010][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:54,336][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:54,661][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:54,986][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:55,310][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:55,634][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:55,959][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:56,284][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:56,608][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:56,934][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:57,260][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:57,585][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:57,911][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:58,240][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:58,567][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:58,894][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:59,219][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:50:59,549][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:00,272][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:01,001][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:01,003][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:01,005][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:01,992][__main__][INFO] - Iteration 289 took 24s (42.25% Gen, 53.73% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 43m 1s. Estimated total time: 20h 30m 15s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 0s, 500 more iterations: 3h 25m 2s.
[2025-11-13 09:51:01,995][__main__][INFO] - Starting iteration 289.
[2025-11-13 09:51:01,999][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:51:01,999][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:51:13,096][__main__][INFO] - Number of regex retries in iteration 289: 0
[2025-11-13 09:51:13,096][__main__][INFO] - agents played in iteration 289 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:51:13,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,684][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:51:13,684][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:51:14,426][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:51:14,723][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:51:15,048][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:51:15,373][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:51:15,700][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:51:16,026][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:51:16,350][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:51:16,676][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:51:17,004][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:51:17,334][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:51:17,662][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:51:17,997][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:51:18,325][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:51:18,650][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:51:18,976][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:51:19,302][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:51:19,626][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:51:19,953][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:51:20,277][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:51:20,603][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:51:20,928][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:51:21,255][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:51:21,581][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:51:21,906][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:51:22,231][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:51:22,556][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:51:22,883][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:51:23,210][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:51:23,542][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:51:23,870][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:51:24,202][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:51:24,522][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:51:24,848][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:25,616][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:26,325][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:26,327][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:26,328][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:27,232][__main__][INFO] - Iteration 290 took 25s (43.97% Gen, 52.44% Train). Generation: 11s, Training: 13s. Estimated remaining time: 19h 14m 4s. Estimated total time: 21h 1m 42s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 3s, 500 more iterations: 3h 30m 17s.
[2025-11-13 09:51:27,233][__main__][INFO] - Starting iteration 290.
[2025-11-13 09:51:27,236][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:51:27,239][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:51:37,718][__main__][INFO] - Number of regex retries in iteration 290: 0
[2025-11-13 09:51:37,718][__main__][INFO] - agents played in iteration 290 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:51:38,206][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,308][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:51:38,309][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:51:39,092][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:51:39,389][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:51:39,714][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:51:40,041][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:51:40,367][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:51:40,693][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:51:41,017][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:51:41,342][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:51:41,667][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:51:41,992][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:51:42,318][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:51:42,641][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:51:42,971][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:51:43,296][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:51:43,620][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:51:43,945][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:51:44,272][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:51:44,597][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:51:44,924][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:51:45,251][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:51:45,583][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:51:45,909][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:51:46,235][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:51:46,560][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:51:46,897][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:51:47,223][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:51:47,549][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:51:47,874][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:51:48,202][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:51:48,528][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:51:48,854][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:51:49,178][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:51:49,507][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:50,255][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:50,977][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:51,140][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:51,142][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:53,001][__main__][INFO] - Iteration 291 took 25s (40.67% Gen, 52.10% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 40m 13s. Estimated total time: 21h 28m 17s. Time estimates for 10 more iterations: 4m 17s, 100 more iterations: 42m 56s, 500 more iterations: 3h 34m 42s.
[2025-11-13 09:51:53,004][__main__][INFO] - Starting iteration 291.
[2025-11-13 09:51:53,006][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:51:53,007][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:03,586][__main__][INFO] - Number of regex retries in iteration 291: 0
[2025-11-13 09:52:03,587][__main__][INFO] - agents played in iteration 291 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:52:04,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:04,111][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:04,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:04,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:04,179][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:04,179][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:04,911][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:05,208][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:05,534][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:05,861][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:06,188][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:06,522][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:06,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:07,170][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:07,496][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:07,827][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:08,151][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:08,478][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:08,805][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:09,132][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:09,461][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:09,788][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:10,113][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:10,441][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:10,767][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:11,094][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:52:11,420][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:52:11,745][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:52:12,072][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:52:12,397][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:52:12,724][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:52:13,053][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:52:13,379][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:52:13,707][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:52:14,035][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:52:14,360][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:52:14,687][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:52:15,014][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:52:15,342][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:52:16,120][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:52:16,824][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:52:16,826][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:52:16,828][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:52:17,810][__main__][INFO] - Iteration 292 took 24s (42.65% Gen, 53.39% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 51m 44s. Estimated total time: 20h 40m 13s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 20s, 500 more iterations: 3h 26m 42s.
[2025-11-13 09:52:17,812][__main__][INFO] - Starting iteration 292.
[2025-11-13 09:52:17,814][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:52:17,815][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:27,835][__main__][INFO] - Number of regex retries in iteration 292: 0
[2025-11-13 09:52:27,836][__main__][INFO] - agents played in iteration 292 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:52:28,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:28,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:28,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:28,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:28,395][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:28,396][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:29,158][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:29,454][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:29,783][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:30,117][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:30,433][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:30,758][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:31,082][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:31,414][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:31,730][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:32,055][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:32,382][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:32,712][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:33,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:33,367][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:33,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:34,020][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:34,346][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:34,672][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:34,998][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:52:35,651][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:52:35,979][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:52:36,306][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:52:36,632][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:52:36,958][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:52:37,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:52:37,608][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:52:37,934][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:52:38,260][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:52:38,586][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:52:38,911][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:52:39,236][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:52:39,562][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:52:40,335][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:52:41,052][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:52:41,054][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:52:41,055][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:52:41,976][__main__][INFO] - Iteration 293 took 24s (41.47% Gen, 54.71% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 19m 13s. Estimated total time: 20h 8m 6s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 16s, 500 more iterations: 3h 21m 21s.
[2025-11-13 09:52:41,978][__main__][INFO] - Starting iteration 293.
[2025-11-13 09:52:41,981][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:52:41,981][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:52,363][__main__][INFO] - Number of regex retries in iteration 293: 0
[2025-11-13 09:52:52,364][__main__][INFO] - agents played in iteration 293 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:52:52,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:52,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:52,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:52,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:52,956][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:52,956][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:53,699][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:53,996][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:54,322][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:54,647][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:54,972][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:55,300][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:55,628][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:55,956][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:56,284][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:56,612][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:56,940][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:57,266][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:57,594][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:57,921][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:58,247][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:58,575][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:58,907][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:59,235][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:59,561][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:59,885][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:53:00,211][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:53:00,538][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:53:00,863][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:53:01,189][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:53:01,514][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:53:01,843][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:53:02,165][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:53:02,491][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:53:02,818][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:03,144][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:03,470][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:03,798][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:04,125][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:04,905][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:05,627][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:05,629][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:05,630][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:06,539][__main__][INFO] - Iteration 294 took 24s (42.27% Gen, 54.02% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 38m 40s. Estimated total time: 20h 27m 57s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 55s, 500 more iterations: 3h 24m 39s.
[2025-11-13 09:53:06,541][__main__][INFO] - Starting iteration 294.
[2025-11-13 09:53:06,544][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:06,544][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:16,320][__main__][INFO] - Number of regex retries in iteration 294: 0
[2025-11-13 09:53:16,321][__main__][INFO] - agents played in iteration 294 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:53:16,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:16,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:16,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:16,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:16,900][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:53:16,901][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:53:17,672][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:53:17,968][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:53:18,293][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:53:18,618][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:53:18,943][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:53:19,270][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:53:19,598][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:53:19,925][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:53:20,252][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:53:20,578][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:53:20,903][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:53:21,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:53:21,556][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:53:21,885][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:53:22,212][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:53:22,539][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:53:22,865][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:53:23,191][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:53:23,518][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:53:23,844][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:53:24,170][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:53:24,496][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:53:24,823][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:53:25,148][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:53:25,473][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:53:25,805][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:53:26,133][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:53:26,458][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:53:26,784][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:27,121][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:27,446][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:28,099][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:28,865][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:29,603][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:29,604][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:29,609][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:30,545][__main__][INFO] - Iteration 295 took 24s (40.73% Gen, 55.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 10m 23s. Estimated total time: 20h 0m 4s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 0s.
[2025-11-13 09:53:30,547][__main__][INFO] - Starting iteration 295.
[2025-11-13 09:53:30,550][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:30,550][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:40,757][__main__][INFO] - Number of regex retries in iteration 295: 0
[2025-11-13 09:53:40,758][__main__][INFO] - agents played in iteration 295 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:53:41,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:41,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:41,311][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:41,344][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:41,345][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:53:41,345][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:53:42,047][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:53:42,348][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:53:42,666][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:53:42,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:53:43,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:53:43,648][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:53:43,969][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:53:44,296][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:53:44,622][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:53:44,948][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:53:45,275][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:53:45,601][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:53:45,929][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:53:46,256][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:53:46,581][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:53:46,906][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:53:47,232][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:53:47,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:53:47,884][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:53:48,211][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:53:48,537][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:53:48,868][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:53:49,193][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:53:49,520][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:53:49,846][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:53:50,172][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:53:50,502][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:53:50,830][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:53:51,157][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:51,483][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:51,813][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:52,139][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:52,466][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:53,238][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:53,940][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:53,942][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:53,944][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:54,869][__main__][INFO] - Iteration 296 took 24s (41.97% Gen, 54.22% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 25m 54s. Estimated total time: 20h 16m 0s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 32s, 500 more iterations: 3h 22m 40s.
[2025-11-13 09:53:54,871][__main__][INFO] - Starting iteration 296.
[2025-11-13 09:53:54,874][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:54,874][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:04,173][__main__][INFO] - Number of regex retries in iteration 296: 0
[2025-11-13 09:54:04,174][__main__][INFO] - agents played in iteration 296 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:54:04,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:04,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:04,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:04,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:04,776][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:04,776][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:05,514][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:05,808][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:06,132][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:06,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:06,788][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:07,117][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:07,449][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:07,768][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:08,095][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:08,752][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:09,072][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:09,403][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:09,732][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:10,062][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:10,383][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:10,708][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:11,035][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:11,360][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:11,685][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:12,011][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:12,337][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:12,665][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:54:12,989][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:54:13,317][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:54:13,642][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:54:13,969][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:54:14,295][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:54:14,622][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:54:14,948][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:54:15,275][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:54:15,600][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:54:15,926][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:54:16,705][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:54:17,425][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:17,427][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:17,428][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:54:18,429][__main__][INFO] - Iteration 297 took 23s (39.48% Gen, 56.26% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 47m 19s. Estimated total time: 19h 37m 49s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 18s.
[2025-11-13 09:54:18,431][__main__][INFO] - Starting iteration 297.
[2025-11-13 09:54:18,434][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:54:18,435][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:27,953][__main__][INFO] - Number of regex retries in iteration 297: 0
[2025-11-13 09:54:27,954][__main__][INFO] - agents played in iteration 297 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:54:28,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:28,483][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:28,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:28,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:28,554][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:28,555][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:29,304][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:29,601][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:29,930][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:30,256][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:30,584][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:30,911][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:31,241][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:31,568][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:31,898][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:32,233][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:32,557][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:32,881][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:33,209][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:33,542][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:33,866][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:34,191][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:34,516][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:34,842][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:35,169][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:35,496][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:35,823][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:36,156][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:36,475][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:54:36,801][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:54:37,128][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:54:37,460][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:54:37,787][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:54:38,113][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:54:38,440][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:54:38,767][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:54:39,092][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:54:39,423][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:54:39,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:54:40,508][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:54:41,211][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:41,212][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:41,214][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:54:42,160][__main__][INFO] - Iteration 298 took 23s (40.12% Gen, 55.88% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 55m 28s. Estimated total time: 19h 46m 21s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 32s, 500 more iterations: 3h 17m 43s.
[2025-11-13 09:54:42,163][__main__][INFO] - Starting iteration 298.
[2025-11-13 09:54:42,169][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:54:42,169][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:51,512][__main__][INFO] - Number of regex retries in iteration 298: 0
[2025-11-13 09:54:51,513][__main__][INFO] - agents played in iteration 298 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:54:51,988][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:52,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:52,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:52,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:52,088][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:52,088][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:52,823][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:53,120][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:53,447][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:53,772][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:54,098][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:54,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:54,760][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:55,079][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:55,403][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:55,729][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:56,068][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:56,389][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:56,717][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:57,042][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:57,369][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:57,693][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:58,021][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:58,348][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:58,673][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:58,999][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:59,325][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:59,653][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:59,981][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:55:00,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:55:00,635][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:55:00,961][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:55:01,292][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:55:01,635][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:55:01,963][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:55:02,289][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:55:02,615][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:55:02,943][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:55:03,269][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:55:04,019][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:55:04,746][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:55:04,748][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:55:04,750][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:05,694][__main__][INFO] - Iteration 299 took 23s (39.71% Gen, 56.26% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 45m 10s. Estimated total time: 19h 36m 27s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 4s.
[2025-11-13 09:55:05,696][__main__][INFO] - Starting iteration 299.
[2025-11-13 09:55:05,699][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:55:05,700][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:15,109][__main__][INFO] - Number of regex retries in iteration 299: 0
[2025-11-13 09:55:15,110][__main__][INFO] - agents played in iteration 299 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:55:15,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:15,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:15,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:15,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:15,698][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:15,699][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:16,471][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:55:16,768][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:55:17,095][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:55:17,423][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:55:17,752][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:55:18,082][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:55:18,410][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:55:18,745][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:55:19,070][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:55:19,395][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:55:19,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:55:20,059][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:55:20,386][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:55:20,710][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:55:21,036][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:55:21,364][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:55:21,691][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:55:22,019][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:55:22,344][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:55:22,671][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:55:22,999][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:55:23,325][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:55:23,651][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:55:23,976][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:55:24,304][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:55:24,631][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:55:24,958][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:55:25,285][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:55:25,614][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:55:25,944][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:55:26,270][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:55:26,601][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:55:26,928][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:55:27,722][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:55:28,476][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:55:28,478][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:55:28,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:29,511][__main__][INFO] - Iteration 300 took 23s (39.52% Gen, 56.15% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 58m 58s. Estimated total time: 19h 50m 39s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 26s.
[2025-11-13 09:55:29,514][__main__][INFO] - Starting iteration 300.
[2025-11-13 09:55:29,517][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:55:29,518][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:39,546][__main__][INFO] - Number of regex retries in iteration 300: 0
[2025-11-13 09:55:39,547][__main__][INFO] - agents played in iteration 300 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:55:40,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:40,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:40,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:40,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:40,147][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:40,148][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:40,897][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:55:41,192][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:55:41,521][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:55:41,852][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:55:42,179][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:55:42,511][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:55:42,837][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:55:43,162][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:55:43,490][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:55:43,815][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:55:44,140][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:55:44,465][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:55:44,790][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:55:45,115][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:55:45,451][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:55:45,775][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:55:46,101][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:55:46,427][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:55:46,763][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:55:47,089][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:55:47,415][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:55:47,740][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:55:48,070][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:55:48,395][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:55:48,723][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:55:49,049][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:55:49,378][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:55:49,704][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:55:50,031][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:55:50,363][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:55:50,686][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:55:51,013][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:55:51,340][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:55:52,116][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:55:52,816][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:55:52,818][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:55:52,820][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:54,627][__main__][INFO] - Iteration 301 took 25s (39.94% Gen, 52.86% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 3m 27s. Estimated total time: 20h 55m 33s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 51s, 500 more iterations: 3h 29m 15s.
[2025-11-13 09:55:54,629][__main__][INFO] - Starting iteration 301.
[2025-11-13 09:55:54,632][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:55:54,633][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:00,734][mllm.models.large_language_model_local][WARNING] - Response >A< did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 09:56:04,714][__main__][INFO] - Number of regex retries in iteration 301: 1
[2025-11-13 09:56:04,714][__main__][INFO] - agents played in iteration 301 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:56:05,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:05,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:05,254][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:05,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:05,631][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:05,631][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:06,357][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:06,655][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:06,981][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:07,307][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:07,633][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:07,959][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:08,285][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:08,611][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:08,939][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:09,264][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:09,591][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:09,917][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:10,254][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:10,581][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:10,908][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:11,233][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:11,569][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:11,896][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:12,222][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:12,549][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:12,876][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:13,205][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:13,532][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:56:13,860][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:56:14,186][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:56:14,513][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:56:14,839][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:56:15,166][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:56:15,492][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:56:15,819][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:56:16,146][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:56:16,474][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:56:16,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:56:17,559][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:56:18,246][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:18,248][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:18,252][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:19,148][__main__][INFO] - Iteration 302 took 24s (41.12% Gen, 55.22% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 33m 21s. Estimated total time: 20h 25m 51s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 51s, 500 more iterations: 3h 24m 18s.
[2025-11-13 09:56:19,150][__main__][INFO] - Starting iteration 302.
[2025-11-13 09:56:19,154][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:56:19,154][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:28,024][__main__][INFO] - Number of regex retries in iteration 302: 0
[2025-11-13 09:56:28,025][__main__][INFO] - agents played in iteration 302 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:56:28,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:28,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:28,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:28,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:28,633][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:28,633][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:29,377][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:29,673][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:29,999][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:30,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:30,649][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:30,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:31,304][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:31,626][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:31,951][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:32,277][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:32,602][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:32,928][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:33,253][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:33,577][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:33,905][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:34,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:34,557][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:34,887][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:35,215][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:35,540][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:35,867][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:36,193][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:36,524][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:56:36,849][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:56:37,175][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:56:37,501][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:56:37,827][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:56:38,153][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:56:38,479][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:56:38,805][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:56:39,131][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:56:39,463][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:56:39,791][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:56:40,559][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:56:41,264][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:41,265][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:41,267][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:42,187][__main__][INFO] - Iteration 303 took 23s (38.51% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 18m 50s. Estimated total time: 19h 11m 44s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 57s.
[2025-11-13 09:56:42,189][__main__][INFO] - Starting iteration 303.
[2025-11-13 09:56:42,192][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:56:42,192][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:51,460][__main__][INFO] - Number of regex retries in iteration 303: 0
[2025-11-13 09:56:51,460][__main__][INFO] - agents played in iteration 303 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:56:51,942][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:51,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:52,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:52,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:52,044][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:52,045][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:52,788][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:53,084][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:53,411][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:53,736][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:54,067][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:54,395][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:54,721][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:55,049][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:55,377][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:55,702][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:56,028][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:56,353][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:56,679][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:57,007][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:57,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:57,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:57,985][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:58,312][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:58,637][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:58,963][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:59,292][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:59,617][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:59,944][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:57:00,270][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:57:00,602][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:57:00,927][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:57:01,252][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:57:01,579][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:57:01,908][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:57:02,235][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:57:02,561][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:57:02,887][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:57:03,216][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:57:03,990][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:57:04,716][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:04,718][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:04,720][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:05,714][__main__][INFO] - Iteration 304 took 23s (39.39% Gen, 56.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 42m 53s. Estimated total time: 19h 36m 10s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 1s.
[2025-11-13 09:57:05,716][__main__][INFO] - Starting iteration 304.
[2025-11-13 09:57:05,720][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:57:05,720][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:15,482][__main__][INFO] - Number of regex retries in iteration 304: 0
[2025-11-13 09:57:15,483][__main__][INFO] - agents played in iteration 304 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:57:15,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:16,026][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:16,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:16,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:16,092][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:16,092][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:16,854][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:57:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:57:17,480][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:57:17,805][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:57:18,132][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:57:18,458][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:57:18,787][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:57:19,114][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:57:19,439][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:57:19,767][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:57:20,095][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:57:20,420][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:57:20,746][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:57:21,073][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:57:21,399][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:57:21,729][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:57:22,056][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:57:22,384][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:57:22,715][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:57:23,042][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:57:23,368][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:57:23,695][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:57:24,021][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:57:24,347][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:57:24,674][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:57:24,999][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:57:25,326][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:57:25,652][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:57:25,980][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:57:26,306][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:57:26,633][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:57:26,960][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:57:27,287][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:57:28,025][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:57:28,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:28,768][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:28,769][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:29,768][__main__][INFO] - Iteration 305 took 24s (40.59% Gen, 55.25% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 8m 46s. Estimated total time: 20h 2m 27s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 4s, 500 more iterations: 3h 20m 24s.
[2025-11-13 09:57:29,770][__main__][INFO] - Starting iteration 305.
[2025-11-13 09:57:29,773][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:57:29,774][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:38,998][__main__][INFO] - Number of regex retries in iteration 305: 0
[2025-11-13 09:57:38,999][__main__][INFO] - agents played in iteration 305 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:57:39,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:39,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:39,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:39,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:39,582][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:39,582][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:40,334][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:57:40,632][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:57:40,959][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:57:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:57:41,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:57:41,942][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:57:42,270][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:57:42,597][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:57:42,925][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:57:43,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:57:43,581][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:57:43,907][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:57:44,234][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:57:44,561][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:57:44,898][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:57:45,214][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:57:45,540][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:57:45,866][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:57:46,192][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:57:46,517][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:57:46,843][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:57:47,170][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:57:47,497][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:57:47,824][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:57:48,150][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:57:48,476][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:57:48,803][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:57:49,129][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:57:49,455][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:57:49,783][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:57:50,110][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:57:50,436][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:57:50,761][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:57:51,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:57:52,275][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:52,276][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:52,278][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:53,184][__main__][INFO] - Iteration 306 took 23s (39.40% Gen, 56.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 36m 31s. Estimated total time: 19h 30m 35s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 1s, 500 more iterations: 3h 15m 5s.
[2025-11-13 09:57:53,186][__main__][INFO] - Starting iteration 306.
[2025-11-13 09:57:53,189][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:57:53,189][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:02,886][__main__][INFO] - Number of regex retries in iteration 306: 0
[2025-11-13 09:58:02,887][__main__][INFO] - agents played in iteration 306 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:58:03,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:03,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:03,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:03,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:03,495][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:03,496][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:04,248][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:58:04,545][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:58:04,870][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:58:05,197][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:58:05,525][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:58:05,852][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:58:06,177][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:58:06,502][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:58:06,827][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:58:07,153][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:58:07,480][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:58:07,806][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:58:08,132][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:58:08,460][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:58:08,786][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:09,112][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:09,436][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:09,761][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:10,089][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:10,416][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:10,742][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:11,069][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:11,396][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:11,722][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:12,049][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:12,376][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:12,706][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:58:13,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:58:13,358][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:58:13,683][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:58:14,010][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:58:14,338][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:58:14,665][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:58:15,417][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:58:16,156][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:16,157][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:16,159][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:17,098][__main__][INFO] - Iteration 307 took 23s (40.56% Gen, 55.51% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 1m 2s. Estimated total time: 19h 55m 30s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 51s, 500 more iterations: 3h 19m 15s.
[2025-11-13 09:58:17,100][__main__][INFO] - Starting iteration 307.
[2025-11-13 09:58:17,184][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:58:17,184][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:26,654][__main__][INFO] - Number of regex retries in iteration 307: 0
[2025-11-13 09:58:26,654][__main__][INFO] - agents played in iteration 307 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:58:27,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:27,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:27,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:27,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:27,249][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:27,249][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:28,001][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:58:28,298][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:58:28,624][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:58:28,949][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:58:29,278][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:58:29,606][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:58:29,930][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:58:30,255][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:58:30,581][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:58:30,908][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:58:31,235][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:58:31,563][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:58:31,896][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:58:32,215][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:58:32,542][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:32,867][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:33,195][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:33,521][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:33,851][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:34,177][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:34,503][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:34,832][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:35,158][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:35,484][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:35,811][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:36,137][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:36,463][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:58:36,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:58:37,117][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:58:37,452][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:58:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:58:38,104][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:58:38,431][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:58:39,177][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:58:39,921][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:39,923][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:39,924][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:40,847][__main__][INFO] - Iteration 308 took 23s (39.88% Gen, 55.89% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 52m 21s. Estimated total time: 19h 47m 13s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 34s, 500 more iterations: 3h 17m 52s.
[2025-11-13 09:58:40,849][__main__][INFO] - Starting iteration 308.
[2025-11-13 09:58:40,853][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:58:40,853][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:50,135][__main__][INFO] - Number of regex retries in iteration 308: 0
[2025-11-13 09:58:50,135][__main__][INFO] - agents played in iteration 308 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:58:50,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:50,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:50,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:50,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:50,762][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:50,763][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:51,506][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:58:51,802][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:58:52,128][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:58:52,455][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:58:52,781][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:58:53,110][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:58:53,436][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:58:53,762][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:58:54,088][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:58:54,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:58:54,738][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:58:55,064][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:58:55,391][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:58:55,719][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:58:56,045][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:56,370][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:56,699][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:57,025][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:57,350][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:57,676][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:58,002][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:58,334][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:58,667][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:58,994][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:59,320][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:59,646][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:59,984][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:59:00,308][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:59:00,637][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:59:00,964][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:59:01,289][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:59:01,614][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:59:01,940][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:59:02,680][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:59:03,433][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:03,435][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:03,437][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:04,373][__main__][INFO] - Iteration 309 took 23s (39.46% Gen, 56.55% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 47s. Estimated total time: 19h 36m 2s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 0s.
[2025-11-13 09:59:04,375][__main__][INFO] - Starting iteration 309.
[2025-11-13 09:59:04,378][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:59:04,378][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:13,564][__main__][INFO] - Number of regex retries in iteration 309: 0
[2025-11-13 09:59:13,564][__main__][INFO] - agents played in iteration 309 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:59:14,047][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:14,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:14,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:14,148][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:14,148][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:14,149][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:14,914][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:59:15,209][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:59:15,536][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:59:15,865][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:59:16,192][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:59:16,518][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:59:16,844][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:59:17,174][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:59:17,500][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:59:17,826][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:59:18,152][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:59:18,477][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:59:18,802][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:59:19,129][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:59:19,455][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:59:19,782][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:59:20,108][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:59:20,434][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:59:20,759][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:59:21,087][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:59:21,417][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:59:21,744][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:59:22,070][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:59:22,398][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:59:22,725][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:59:23,051][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:59:23,377][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:59:23,703][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:59:24,027][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:59:24,356][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:59:24,687][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:59:25,008][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:59:25,337][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:59:26,102][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:59:26,835][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:26,836][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:26,838][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:27,879][__main__][INFO] - Iteration 310 took 23s (39.08% Gen, 56.48% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 39m 29s. Estimated total time: 19h 35m 8s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 51s.
[2025-11-13 09:59:27,882][__main__][INFO] - Starting iteration 310.
[2025-11-13 09:59:27,885][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:59:27,885][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:36,857][__main__][INFO] - Number of regex retries in iteration 310: 0
[2025-11-13 09:59:36,858][__main__][INFO] - agents played in iteration 310 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 09:59:37,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:37,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:37,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:37,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:37,449][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:37,450][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:38,204][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:59:38,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:59:38,829][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:59:39,155][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:59:39,487][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:59:39,816][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:59:40,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:59:40,468][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:59:40,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:59:41,121][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:59:41,447][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:59:41,771][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:59:42,099][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:59:42,424][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:59:42,751][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:59:43,077][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:59:43,403][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:59:43,729][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:59:44,056][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:59:44,388][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:59:44,708][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:59:45,034][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:59:45,362][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:59:45,696][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:59:46,018][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:59:46,344][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:59:46,674][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:59:47,001][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:59:47,327][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:59:47,652][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:59:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:59:48,304][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:59:48,630][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:59:49,380][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:59:50,130][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:50,131][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:50,133][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:52,166][__main__][INFO] - Iteration 311 took 24s (36.95% Gen, 54.67% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 18m 1s. Estimated total time: 20h 14m 5s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 28s, 500 more iterations: 3h 22m 20s.
[2025-11-13 09:59:52,168][__main__][INFO] - Starting iteration 311.
[2025-11-13 09:59:52,171][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 09:59:52,172][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:01,036][__main__][INFO] - Number of regex retries in iteration 311: 0
[2025-11-13 10:00:01,036][__main__][INFO] - agents played in iteration 311 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:00:01,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:01,559][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:01,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:01,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:01,627][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:01,628][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:02,403][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:00:02,792][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:00:03,119][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:00:03,444][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:00:03,769][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:00:04,093][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:00:04,419][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:00:04,746][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:00:05,073][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:00:05,398][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:00:05,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:00:06,050][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:00:06,376][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:00:06,701][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:00:07,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:00:07,353][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:00:07,679][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:00:08,006][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:00:08,335][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:00:08,660][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:00:08,985][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:00:09,311][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:00:09,636][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:00:09,963][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:10,291][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:10,617][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:10,942][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:11,269][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:11,595][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:11,925][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:12,252][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:12,577][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:12,908][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:00:13,622][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:00:14,341][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:14,342][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:14,344][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:15,333][__main__][INFO] - Iteration 312 took 23s (38.27% Gen, 57.45% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 21m 41s. Estimated total time: 19h 18m 8s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 1s.
[2025-11-13 10:00:15,335][__main__][INFO] - Starting iteration 312.
[2025-11-13 10:00:15,338][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:00:15,339][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:24,913][__main__][INFO] - Number of regex retries in iteration 312: 0
[2025-11-13 10:00:24,914][__main__][INFO] - agents played in iteration 312 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:00:25,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:25,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:25,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:25,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:25,508][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:25,508][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:26,266][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:00:26,563][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:00:26,899][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:00:27,225][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:00:27,551][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:00:27,876][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:00:28,201][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:00:28,527][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:00:28,852][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:00:29,177][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:00:29,502][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:00:29,828][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:00:30,153][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:00:30,478][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:00:30,805][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:00:31,131][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:00:31,457][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:00:31,783][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:00:32,114][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:00:32,441][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:00:32,766][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:00:33,094][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:00:33,422][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:00:33,748][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:34,074][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:34,401][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:34,727][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:35,051][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:35,377][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:35,702][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:36,027][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:36,355][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:36,684][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:00:37,425][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:00:38,163][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:38,164][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:38,166][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:39,143][__main__][INFO] - Iteration 313 took 23s (40.22% Gen, 55.67% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 53m 25s. Estimated total time: 19h 50m 15s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 40s, 500 more iterations: 3h 18m 22s.
[2025-11-13 10:00:39,145][__main__][INFO] - Starting iteration 313.
[2025-11-13 10:00:39,148][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:00:39,148][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:47,666][__main__][INFO] - Number of regex retries in iteration 313: 0
[2025-11-13 10:00:47,667][__main__][INFO] - agents played in iteration 313 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:00:48,148][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:48,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:48,215][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:48,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:48,249][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:48,250][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:49,030][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:00:49,327][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:00:49,654][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:00:49,980][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:00:50,307][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:00:50,632][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:00:50,959][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:00:51,285][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:00:51,612][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:00:51,937][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:00:52,263][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:00:52,588][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:00:52,923][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:00:53,244][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:00:53,569][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:00:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:00:54,220][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:00:54,546][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:00:54,871][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:00:55,198][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:00:55,527][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:00:55,850][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:00:56,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:00:56,501][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:56,831][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:57,157][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:57,483][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:57,808][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:58,136][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:58,459][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:58,787][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:59,113][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:59,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:01:00,255][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:01:01,060][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:01,061][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:01,062][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:02,101][__main__][INFO] - Iteration 314 took 22s (37.11% Gen, 58.36% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 10m 28s. Estimated total time: 19h 7m 41s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 16s.
[2025-11-13 10:01:02,104][__main__][INFO] - Starting iteration 314.
[2025-11-13 10:01:02,110][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:01:02,112][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:11,523][__main__][INFO] - Number of regex retries in iteration 314: 0
[2025-11-13 10:01:11,524][__main__][INFO] - agents played in iteration 314 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:01:12,006][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:12,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:12,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:12,108][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:12,109][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:12,109][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:12,880][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:01:13,178][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:01:13,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:01:13,834][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:01:14,159][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:01:14,487][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:01:14,814][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:01:15,141][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:01:15,467][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:01:15,795][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:01:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:01:16,445][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:01:16,771][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:01:17,096][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:01:17,421][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:01:17,747][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:01:18,073][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:01:18,401][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:01:18,728][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:01:19,054][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:01:19,380][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:01:19,707][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:01:20,035][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:01:20,364][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:01:20,691][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:01:21,018][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:01:21,350][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:01:21,680][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:01:22,009][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:01:22,340][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:01:22,665][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:01:22,990][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:01:23,316][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:01:24,038][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:01:24,761][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:24,763][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:24,764][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:25,772][__main__][INFO] - Iteration 315 took 23s (39.77% Gen, 55.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 45m 34s. Estimated total time: 19h 43m 11s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 26s, 500 more iterations: 3h 17m 11s.
[2025-11-13 10:01:25,774][__main__][INFO] - Starting iteration 315.
[2025-11-13 10:01:25,778][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:01:25,778][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:35,622][__main__][INFO] - Number of regex retries in iteration 315: 0
[2025-11-13 10:01:35,623][__main__][INFO] - agents played in iteration 315 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:01:36,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:36,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:36,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:36,219][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:36,219][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:36,219][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:36,986][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:01:37,284][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:01:37,614][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:01:37,939][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:01:38,264][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:01:38,591][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:01:38,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:01:39,241][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:01:39,567][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:01:39,892][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:01:40,225][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:01:40,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:01:40,875][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:01:41,203][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:01:41,535][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:01:41,855][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:01:42,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:01:42,504][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:01:42,831][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:01:43,157][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:01:43,483][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:01:43,809][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:01:44,134][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:01:44,463][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:01:44,789][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:01:45,113][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:01:45,438][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:01:45,764][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:01:46,089][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:01:46,413][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:01:46,739][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:01:47,066][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:01:47,391][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:01:48,132][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:01:48,873][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:48,875][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:48,876][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:49,871][__main__][INFO] - Iteration 316 took 24s (40.86% Gen, 55.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 6m 41s. Estimated total time: 20h 4m 42s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 9s, 500 more iterations: 3h 20m 47s.
[2025-11-13 10:01:49,873][__main__][INFO] - Starting iteration 316.
[2025-11-13 10:01:49,876][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:01:49,877][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:59,559][__main__][INFO] - Number of regex retries in iteration 316: 0
[2025-11-13 10:01:59,559][__main__][INFO] - agents played in iteration 316 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:02:00,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:00,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:00,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:00,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:00,158][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:00,159][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:00,940][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:01,237][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:01,563][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:01,889][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:02,215][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:02:02,541][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:02:02,867][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:02:03,195][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:02:03,521][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:02:03,846][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:02:04,171][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:02:04,497][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:02:04,823][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:02:05,150][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:02:05,480][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:02:05,804][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:02:06,129][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:02:06,455][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:02:06,781][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:02:07,107][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:02:07,438][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:02:07,763][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:02:08,090][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:02:08,415][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:02:08,740][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:02:09,067][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:02:09,392][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:02:09,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:02:10,042][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:02:10,367][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:02:10,693][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:02:11,018][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:02:11,343][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:02:12,129][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:02:12,873][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:12,874][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:12,876][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:13,890][__main__][INFO] - Iteration 317 took 24s (40.32% Gen, 55.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 2m 18s. Estimated total time: 20h 0m 43s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 1s, 500 more iterations: 3h 20m 7s.
[2025-11-13 10:02:13,892][__main__][INFO] - Starting iteration 317.
[2025-11-13 10:02:13,895][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:02:13,896][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:02:23,671][__main__][INFO] - Number of regex retries in iteration 317: 0
[2025-11-13 10:02:23,671][__main__][INFO] - agents played in iteration 317 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:02:24,163][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:24,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:24,230][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:24,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:24,263][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:24,264][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:25,032][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:25,329][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:25,980][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:26,310][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:02:26,634][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:02:26,960][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:02:27,285][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:02:27,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:02:27,941][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:02:28,267][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:02:28,592][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:02:28,916][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:02:29,241][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:02:29,567][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:02:29,893][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:02:30,221][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:02:30,548][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:02:30,874][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:02:31,200][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:02:31,528][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:02:31,854][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:02:32,182][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:02:32,507][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:02:32,833][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:02:33,158][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:02:33,484][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:02:33,811][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:02:34,138][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:02:34,464][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:02:34,795][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:02:35,121][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:02:35,446][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:02:36,183][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:02:36,924][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:36,926][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:36,927][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:37,919][__main__][INFO] - Iteration 318 took 24s (40.69% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 2m 25s. Estimated total time: 20h 1m 14s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 2s, 500 more iterations: 3h 20m 12s.
[2025-11-13 10:02:37,922][__main__][INFO] - Starting iteration 318.
[2025-11-13 10:02:37,928][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:02:37,929][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:02:47,598][__main__][INFO] - Number of regex retries in iteration 318: 0
[2025-11-13 10:02:47,598][__main__][INFO] - agents played in iteration 318 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:02:48,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:48,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:48,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:48,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:48,195][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:48,196][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:48,982][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:49,278][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:49,605][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:49,934][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:50,261][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:02:50,585][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:02:50,911][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:02:51,238][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:02:51,563][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:02:51,891][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:02:52,217][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:02:52,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:02:52,872][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:02:53,198][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:02:53,523][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:02:53,851][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:02:54,181][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:02:54,507][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:02:54,833][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:02:55,159][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:02:55,484][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:02:55,809][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:02:56,135][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:02:56,462][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:02:56,788][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:02:57,116][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:02:57,443][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:02:57,768][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:02:58,094][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:02:58,421][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:02:58,746][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:02:59,070][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:02:59,396][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:00,122][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:00,865][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:00,866][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:00,868][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:01,905][__main__][INFO] - Iteration 319 took 23s (40.32% Gen, 55.34% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 59m 42s. Estimated total time: 19h 58m 56s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 57s, 500 more iterations: 3h 19m 49s.
[2025-11-13 10:03:01,907][__main__][INFO] - Starting iteration 319.
[2025-11-13 10:03:01,910][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:03:01,911][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:03:11,339][__main__][INFO] - Number of regex retries in iteration 319: 0
[2025-11-13 10:03:11,339][__main__][INFO] - agents played in iteration 319 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:03:11,843][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:11,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:11,910][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:11,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:11,945][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:03:11,945][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:03:12,726][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:03:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:03:13,348][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:03:13,673][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:03:14,000][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:03:14,329][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:03:14,656][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:03:14,981][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:03:15,307][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:03:15,634][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:03:15,960][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:03:16,287][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:03:16,613][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:03:16,939][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:03:17,268][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:03:17,591][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:03:17,920][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:03:18,249][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:03:18,575][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:03:18,903][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:03:19,237][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:03:19,564][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:03:19,892][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:03:20,218][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:03:20,548][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:03:20,873][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:03:21,203][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:03:21,530][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:03:21,854][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:03:22,184][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:03:22,513][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:03:22,847][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:03:23,165][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:23,909][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:24,647][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:24,648][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:24,650][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:25,645][__main__][INFO] - Iteration 320 took 23s (39.72% Gen, 56.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 47m 9s. Estimated total time: 19h 46m 45s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 33s, 500 more iterations: 3h 17m 47s.
[2025-11-13 10:03:25,647][__main__][INFO] - Starting iteration 320.
[2025-11-13 10:03:25,650][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:03:25,650][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:03:34,915][__main__][INFO] - Number of regex retries in iteration 320: 0
[2025-11-13 10:03:34,916][__main__][INFO] - agents played in iteration 320 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:03:35,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:35,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:35,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:35,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:35,590][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:03:35,590][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:03:36,390][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:03:36,688][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:03:37,015][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:03:37,341][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:03:37,667][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:03:37,992][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:03:38,317][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:03:38,650][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:03:38,973][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:03:39,298][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:03:39,624][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:03:39,959][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:03:40,278][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:03:40,603][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:03:40,928][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:03:41,259][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:03:41,581][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:03:41,910][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:03:42,236][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:03:42,561][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:03:42,887][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:03:43,210][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:03:43,537][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:03:43,862][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:03:44,189][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:03:44,518][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:03:44,848][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:03:45,176][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:03:45,501][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:03:45,826][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:03:46,151][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:03:46,478][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:03:46,803][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:47,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:48,256][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:48,258][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:48,260][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:50,316][__main__][INFO] - Iteration 321 took 24s (37.56% Gen, 54.10% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 33m 20s. Estimated total time: 20h 33m 21s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 6s, 500 more iterations: 3h 25m 33s.
[2025-11-13 10:03:50,319][__main__][INFO] - Starting iteration 321.
[2025-11-13 10:03:50,322][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:03:50,322][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:00,474][__main__][INFO] - Number of regex retries in iteration 321: 0
[2025-11-13 10:04:00,475][__main__][INFO] - agents played in iteration 321 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:04:00,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:01,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:01,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:01,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:01,084][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:01,084][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:01,841][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:02,137][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:02,469][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:02,790][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:03,117][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:03,442][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:03,774][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:04,100][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:04,429][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:04,755][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:05,086][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:05,414][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:05,741][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:06,066][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:06,393][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:06,720][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:04:07,048][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:04:07,373][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:04:07,701][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:04:08,032][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:04:08,359][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:04:08,686][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:04:09,012][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:04:09,349][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:04:09,673][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:04:09,997][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:04:10,324][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:04:10,657][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:04:10,983][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:04:11,309][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:04:11,634][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:04:11,964][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:04:12,292][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:04:13,049][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:04:13,777][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:04:13,779][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:04:13,780][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:04:14,732][__main__][INFO] - Iteration 322 took 24s (41.59% Gen, 54.50% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 20m 6s. Estimated total time: 20h 20m 32s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 41s, 500 more iterations: 3h 23m 25s.
[2025-11-13 10:04:14,734][__main__][INFO] - Starting iteration 322.
[2025-11-13 10:04:14,736][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:04:14,737][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:23,897][__main__][INFO] - Number of regex retries in iteration 322: 0
[2025-11-13 10:04:23,897][__main__][INFO] - agents played in iteration 322 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:04:24,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:24,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:24,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:24,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:24,483][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:24,483][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:25,271][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:25,568][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:25,895][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:26,221][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:26,546][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:26,872][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:27,197][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:27,529][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:27,859][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:28,190][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:28,516][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:28,847][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:29,176][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:29,503][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:29,832][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:30,155][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:04:30,484][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:04:30,812][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:04:31,138][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:04:31,465][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:04:31,793][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:04:32,117][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:04:32,444][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:04:32,768][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:04:33,093][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:04:33,420][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:04:33,745][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:04:34,069][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:04:34,393][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:04:34,717][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:04:35,041][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:04:35,375][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:04:35,692][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:04:36,448][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:04:37,223][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:04:37,224][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:04:37,226][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:04:38,202][__main__][INFO] - Iteration 323 took 23s (39.03% Gen, 56.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 32m 31s. Estimated total time: 19h 33m 20s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 6s, 500 more iterations: 3h 15m 33s.
[2025-11-13 10:04:38,205][__main__][INFO] - Starting iteration 323.
[2025-11-13 10:04:38,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:04:38,209][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:47,373][__main__][INFO] - Number of regex retries in iteration 323: 0
[2025-11-13 10:04:47,374][__main__][INFO] - agents played in iteration 323 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:04:47,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:48,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:48,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:48,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:48,333][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:48,333][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:49,115][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:49,412][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:49,738][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:50,063][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:50,388][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:50,713][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:51,038][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:51,368][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:51,694][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:52,020][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:52,351][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:52,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:53,002][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:53,329][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:53,659][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:53,982][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:04:54,309][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:04:54,636][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:04:54,969][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:04:55,286][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:04:55,612][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:04:55,939][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:04:56,273][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:04:56,592][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:04:56,918][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:04:57,243][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:04:57,575][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:04:57,896][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:04:58,220][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:04:58,545][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:04:58,877][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:04:59,195][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:04:59,521][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:00,253][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:00,986][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:00,987][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:00,989][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:01,939][__main__][INFO] - Iteration 324 took 23s (38.62% Gen, 57.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 45m 22s. Estimated total time: 19h 46m 36s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 33s, 500 more iterations: 3h 17m 46s.
[2025-11-13 10:05:01,942][__main__][INFO] - Starting iteration 324.
[2025-11-13 10:05:01,944][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:01,945][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:11,445][__main__][INFO] - Number of regex retries in iteration 324: 0
[2025-11-13 10:05:11,445][__main__][INFO] - agents played in iteration 324 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:05:11,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:11,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:11,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:12,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:12,037][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:12,038][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:12,822][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:05:13,118][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:05:13,445][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:05:13,773][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:05:14,098][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:05:14,424][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:05:14,749][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:05:15,075][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:05:15,401][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:05:15,726][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:05:16,058][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:05:16,385][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:05:16,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:05:17,037][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:05:17,376][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:05:17,695][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:05:18,020][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:05:18,345][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:05:18,678][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:05:18,999][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:05:19,323][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:05:19,649][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:05:19,973][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:05:20,297][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:05:20,622][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:05:20,946][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:05:21,272][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:05:21,596][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:05:21,922][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:05:22,247][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:05:22,572][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:05:22,896][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:05:23,220][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:23,972][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:24,711][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:24,713][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:24,715][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:25,758][__main__][INFO] - Iteration 325 took 23s (39.89% Gen, 55.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 49m 6s. Estimated total time: 19h 50m 43s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 27s.
[2025-11-13 10:05:25,760][__main__][INFO] - Starting iteration 325.
[2025-11-13 10:05:25,763][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:25,763][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:35,437][__main__][INFO] - Number of regex retries in iteration 325: 0
[2025-11-13 10:05:35,438][__main__][INFO] - agents played in iteration 325 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:05:35,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:35,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:36,000][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:36,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:36,034][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:36,035][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:36,823][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:05:37,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:05:37,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:05:37,769][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:05:38,097][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:05:38,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:05:38,748][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:05:39,072][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:05:39,399][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:05:39,725][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:05:40,053][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:05:40,379][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:05:40,705][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:05:41,030][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:05:41,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:05:41,691][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:05:42,017][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:05:42,342][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:05:42,667][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:05:42,991][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:05:43,315][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:05:43,640][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:05:43,964][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:05:44,288][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:05:44,614][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:05:44,939][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:05:45,267][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:05:45,596][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:05:45,921][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:05:46,251][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:05:46,577][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:05:46,902][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:05:47,227][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:47,970][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:48,697][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:48,700][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:48,701][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:49,673][__main__][INFO] - Iteration 326 took 23s (40.46% Gen, 55.47% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 53m 33s. Estimated total time: 19h 55m 34s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 51s, 500 more iterations: 3h 19m 15s.
[2025-11-13 10:05:49,675][__main__][INFO] - Starting iteration 326.
[2025-11-13 10:05:49,678][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:49,678][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:58,257][__main__][INFO] - Number of regex retries in iteration 326: 0
[2025-11-13 10:05:58,258][__main__][INFO] - agents played in iteration 326 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:05:58,745][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:59,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:59,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:59,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:59,215][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:59,215][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:59,988][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:00,286][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:00,611][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:00,938][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:01,265][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:01,590][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:01,917][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:02,246][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:02,577][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:02,907][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:03,234][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:03,561][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:03,891][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:04,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:04,549][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:04,873][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:05,201][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:05,526][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:05,852][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:06,181][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:06:06,506][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:06:06,830][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:06:07,156][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:06:07,480][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:06:07,804][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:06:08,129][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:06:08,454][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:06:08,779][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:06:09,103][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:06:09,427][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:06:09,752][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:06:10,076][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:06:10,401][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:06:11,121][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:06:11,892][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:06:11,893][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:06:11,896][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:06:12,966][__main__][INFO] - Iteration 327 took 23s (36.83% Gen, 58.56% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 22m 4s. Estimated total time: 19h 24m 28s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 4s.
[2025-11-13 10:06:12,968][__main__][INFO] - Starting iteration 327.
[2025-11-13 10:06:12,972][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:06:12,972][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:23,064][__main__][INFO] - Number of regex retries in iteration 327: 0
[2025-11-13 10:06:23,065][__main__][INFO] - agents played in iteration 327 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:06:23,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:23,582][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:23,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:23,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:23,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:23,650][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:24,431][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:24,729][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:25,056][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:25,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:25,712][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:26,043][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:26,370][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:26,696][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:27,022][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:27,349][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:27,676][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:28,003][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:28,329][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:28,654][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:28,982][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:29,313][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:29,646][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:29,979][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:30,307][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:30,632][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:06:30,958][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:06:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:06:31,613][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:06:31,943][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:06:32,267][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:06:32,594][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:06:32,920][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:06:33,245][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:06:33,581][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:06:33,909][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:06:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:06:34,560][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:06:34,888][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:06:35,624][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:06:36,382][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:06:36,383][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:06:36,385][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:06:37,359][__main__][INFO] - Iteration 328 took 24s (41.38% Gen, 54.62% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 16m 37s. Estimated total time: 20h 19m 25s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 38s, 500 more iterations: 3h 23m 14s.
[2025-11-13 10:06:37,364][__main__][INFO] - Starting iteration 328.
[2025-11-13 10:06:37,367][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:06:37,367][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:47,251][__main__][INFO] - Number of regex retries in iteration 328: 0
[2025-11-13 10:06:47,252][__main__][INFO] - agents played in iteration 328 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:06:47,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:47,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:47,808][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:47,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:47,846][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:47,846][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:48,641][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:48,938][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:49,266][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:49,602][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:49,928][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:50,253][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:50,580][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:50,911][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:51,236][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:51,561][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:51,888][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:52,217][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:52,545][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:52,872][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:53,202][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:53,523][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:53,849][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:54,175][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:54,505][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:54,827][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:06:55,151][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:06:55,476][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:06:55,800][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:06:56,126][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:06:56,451][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:06:56,775][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:06:57,100][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:06:57,424][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:06:57,750][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:06:58,075][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:06:58,401][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:06:58,726][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:06:59,052][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:06:59,813][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:00,557][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:00,558][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:00,560][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:01,530][__main__][INFO] - Iteration 329 took 24s (40.91% Gen, 55.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 5m 1s. Estimated total time: 20h 8m 13s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 16s, 500 more iterations: 3h 21m 22s.
[2025-11-13 10:07:01,533][__main__][INFO] - Starting iteration 329.
[2025-11-13 10:07:01,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:07:01,536][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:07:07,084][mllm.models.large_language_model_local][WARNING] - Response <I> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 10:07:10,905][__main__][INFO] - Number of regex retries in iteration 329: 1
[2025-11-13 10:07:10,906][__main__][INFO] - agents played in iteration 329 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:07:11,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:11,442][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:11,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:11,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:11,509][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:07:11,509][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:07:12,276][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:07:12,572][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:07:12,900][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:07:13,226][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:07:13,551][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:07:13,877][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:07:14,209][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:07:14,529][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:07:14,855][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:07:15,179][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:07:15,504][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:07:15,830][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:07:16,159][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:07:16,484][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:07:16,810][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:07:17,141][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:07:17,470][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:07:17,795][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:07:18,119][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:07:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:07:18,767][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:07:19,092][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:07:19,424][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:07:19,741][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:07:20,065][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:07:20,389][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:07:20,715][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:07:21,042][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:07:21,367][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:07:21,693][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:07:22,018][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:07:22,342][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:07:22,670][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:07:23,421][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:24,144][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:24,147][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:24,149][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:25,201][__main__][INFO] - Iteration 330 took 23s (39.59% Gen, 55.96% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 39m 41s. Estimated total time: 19h 43m 18s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 26s, 500 more iterations: 3h 17m 13s.
[2025-11-13 10:07:25,203][__main__][INFO] - Starting iteration 330.
[2025-11-13 10:07:25,206][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:07:25,206][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:07:32,119][mllm.models.large_language_model_local][WARNING] - Response <B>
user
Last round, the other agent played <A>. did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 10:07:34,709][__main__][INFO] - Number of regex retries in iteration 330: 1
[2025-11-13 10:07:34,709][__main__][INFO] - agents played in iteration 330 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:07:35,226][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:35,329][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:07:35,330][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:07:36,114][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:07:36,417][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:07:36,734][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:07:37,060][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:07:37,385][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:07:37,718][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:07:38,035][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:07:38,361][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:07:38,687][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:07:39,018][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:07:39,340][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:07:39,665][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:07:39,992][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:07:40,325][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:07:40,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:07:40,976][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:07:41,301][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:07:41,629][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:07:41,950][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:07:42,276][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:07:42,601][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:07:42,933][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:07:43,256][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:07:43,583][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:07:43,908][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:07:44,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:07:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:07:44,883][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:07:45,209][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:07:45,538][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:07:45,858][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:07:46,183][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:07:46,514][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:07:47,280][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:48,025][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:48,027][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:48,028][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:50,098][__main__][INFO] - Iteration 331 took 24s (38.18% Gen, 53.50% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 40m 40s. Estimated total time: 20h 44m 41s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 29s, 500 more iterations: 3h 27m 26s.
[2025-11-13 10:07:50,101][__main__][INFO] - Starting iteration 331.
[2025-11-13 10:07:50,103][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:07:50,104][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:00,041][__main__][INFO] - Number of regex retries in iteration 331: 0
[2025-11-13 10:08:00,041][__main__][INFO] - agents played in iteration 331 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:08:00,547][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:00,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:00,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:00,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:00,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:00,650][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:01,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:01,708][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:02,034][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:02,360][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:02,687][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:03,015][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:03,339][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:03,664][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:03,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:04,317][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:04,642][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:04,969][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:05,294][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:05,619][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:05,946][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:06,596][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:06,921][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:07,246][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:07,571][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:07,895][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:08,222][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:08,546][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:08,872][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:08:09,197][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:08:09,524][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:08:09,849][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:08:10,174][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:08:10,498][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:08:10,826][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:08:11,150][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:08:11,475][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:08:11,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:08:12,540][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:08:13,280][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:08:13,281][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:08:13,283][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:08:14,259][__main__][INFO] - Iteration 332 took 24s (41.14% Gen, 54.81% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 3m 26s. Estimated total time: 20h 7m 51s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 15s, 500 more iterations: 3h 21m 18s.
[2025-11-13 10:08:14,261][__main__][INFO] - Starting iteration 332.
[2025-11-13 10:08:14,264][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:08:14,265][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:23,838][__main__][INFO] - Number of regex retries in iteration 332: 0
[2025-11-13 10:08:23,839][__main__][INFO] - agents played in iteration 332 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:08:24,333][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:24,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:24,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:24,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:24,433][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:24,433][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:25,199][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:25,496][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:25,822][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:26,149][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:26,474][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:26,799][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:27,125][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:27,451][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:27,777][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:28,114][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:28,441][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:28,766][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:29,092][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:29,419][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:29,745][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:30,072][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:30,400][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:30,731][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:31,054][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:31,379][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:31,707][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:32,037][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:32,364][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:32,691][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:08:33,016][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:08:33,351][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:08:33,676][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:08:34,005][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:08:34,330][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:08:34,656][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:08:34,981][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:08:35,309][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:08:35,635][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:08:36,388][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:08:37,138][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:08:37,141][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:08:37,142][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:08:38,095][__main__][INFO] - Iteration 333 took 23s (40.17% Gen, 55.82% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 45s. Estimated total time: 19h 51m 35s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 35s.
[2025-11-13 10:08:38,097][__main__][INFO] - Starting iteration 333.
[2025-11-13 10:08:38,100][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:08:38,100][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:47,832][__main__][INFO] - Number of regex retries in iteration 333: 0
[2025-11-13 10:08:47,833][__main__][INFO] - agents played in iteration 333 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:08:48,318][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:48,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:48,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:48,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:48,420][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:48,420][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:49,172][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:49,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:49,798][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:50,126][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:50,450][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:50,774][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:51,102][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:51,427][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:51,755][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:52,083][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:52,408][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:52,735][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:53,063][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:53,385][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:53,714][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:54,039][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:54,364][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:54,689][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:55,014][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:55,339][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:55,665][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:56,001][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:56,325][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:56,649][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:08:56,976][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:08:57,300][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:08:57,624][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:08:57,949][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:08:58,275][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:08:58,602][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:08:58,928][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:08:59,259][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:08:59,586][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:00,319][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:01,045][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:01,048][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:01,049][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:02,158][__main__][INFO] - Iteration 334 took 24s (40.45% Gen, 54.93% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 57m 43s. Estimated total time: 20h 2m 56s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 5s, 500 more iterations: 3h 20m 29s.
[2025-11-13 10:09:02,161][__main__][INFO] - Starting iteration 334.
[2025-11-13 10:09:02,164][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:02,165][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:09:12,339][__main__][INFO] - Number of regex retries in iteration 334: 0
[2025-11-13 10:09:12,339][__main__][INFO] - agents played in iteration 334 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:09:12,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:12,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:12,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:12,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:12,929][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:09:12,929][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:09:13,674][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:09:13,972][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:09:14,306][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:09:14,635][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:09:14,963][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:09:15,289][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:09:15,614][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:09:15,942][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:09:16,270][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:09:16,598][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:09:16,923][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:09:17,249][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:09:17,574][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:09:17,899][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:09:18,225][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:09:18,555][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:09:18,878][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:09:19,203][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:09:19,530][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:09:19,856][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:09:20,180][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:09:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:09:20,832][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:09:21,157][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:09:21,482][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:09:21,807][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:09:22,134][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:09:22,459][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:09:22,785][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:09:23,116][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:09:23,446][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:09:23,777][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:09:24,104][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:24,852][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:25,551][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:25,554][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:25,556][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:26,519][__main__][INFO] - Iteration 335 took 24s (41.77% Gen, 54.26% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 12m 11s. Estimated total time: 20h 17m 49s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 35s, 500 more iterations: 3h 22m 58s.
[2025-11-13 10:09:26,521][__main__][INFO] - Starting iteration 335.
[2025-11-13 10:09:26,524][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:26,525][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:09:36,507][__main__][INFO] - Number of regex retries in iteration 335: 0
[2025-11-13 10:09:36,508][__main__][INFO] - agents played in iteration 335 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:09:36,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:37,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:37,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:37,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:37,089][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:09:37,089][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:09:37,839][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:09:38,134][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:09:38,461][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:09:38,788][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:09:39,117][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:09:39,444][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:09:39,770][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:09:40,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:09:40,425][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:09:40,756][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:09:41,085][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:09:41,410][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:09:41,737][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:09:42,062][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:09:42,387][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:09:42,713][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:09:43,040][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:09:43,372][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:09:43,696][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:09:44,020][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:09:44,347][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:09:44,672][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:09:44,998][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:09:45,324][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:09:45,658][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:09:45,976][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:09:46,301][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:09:46,631][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:09:46,962][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:09:47,284][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:09:47,609][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:09:47,938][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:09:48,263][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:49,006][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:49,717][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:49,719][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:49,721][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:50,640][__main__][INFO] - Iteration 336 took 24s (41.39% Gen, 54.79% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 59m 48s. Estimated total time: 20h 5m 49s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 11s, 500 more iterations: 3h 20m 58s.
[2025-11-13 10:09:50,642][__main__][INFO] - Starting iteration 336.
[2025-11-13 10:09:50,645][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:50,646][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:00,222][__main__][INFO] - Number of regex retries in iteration 336: 0
[2025-11-13 10:10:00,223][__main__][INFO] - agents played in iteration 336 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:10:00,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:00,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:00,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:00,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:00,813][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:00,813][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:01,567][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:01,863][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:02,189][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:02,515][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:02,848][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:03,169][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:03,494][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:03,821][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:04,159][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:04,483][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:04,809][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:05,134][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:05,464][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:05,785][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:06,112][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:06,439][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:06,768][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:07,091][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:07,418][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:07,745][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:08,071][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:08,401][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:08,728][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:09,052][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:09,378][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:10:09,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:10:10,039][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:10:10,367][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:10:10,699][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:10:11,037][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:10:11,362][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:10:11,689][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:10:12,017][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:10:12,775][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:10:13,484][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:10:13,486][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:10:13,488][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:10:14,409][__main__][INFO] - Iteration 337 took 23s (40.30% Gen, 55.82% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 41m 50s. Estimated total time: 19h 48m 16s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 36s, 500 more iterations: 3h 18m 2s.
[2025-11-13 10:10:14,411][__main__][INFO] - Starting iteration 337.
[2025-11-13 10:10:14,457][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:10:14,458][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:23,830][__main__][INFO] - Number of regex retries in iteration 337: 0
[2025-11-13 10:10:23,831][__main__][INFO] - agents played in iteration 337 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:10:24,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:24,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:24,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:24,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:24,416][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:24,416][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:25,172][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:25,469][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:25,795][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:26,121][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:26,459][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:26,781][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:27,108][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:27,432][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:27,758][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:28,084][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:28,415][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:28,742][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:29,066][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:29,390][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:29,716][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:30,041][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:30,366][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:30,689][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:31,013][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:31,337][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:31,662][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:31,987][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:32,311][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:32,636][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:32,961][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:10:33,288][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:10:33,614][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:10:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:10:34,263][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:10:34,589][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:10:34,915][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:10:35,244][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:10:35,571][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:10:36,330][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:10:37,021][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:10:37,022][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:10:37,024][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:10:37,996][__main__][INFO] - Iteration 338 took 23s (39.75% Gen, 55.93% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 32m 19s. Estimated total time: 19h 39m 8s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 31s.
[2025-11-13 10:10:37,998][__main__][INFO] - Starting iteration 338.
[2025-11-13 10:10:38,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:10:38,002][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:47,712][__main__][INFO] - Number of regex retries in iteration 338: 0
[2025-11-13 10:10:47,713][__main__][INFO] - agents played in iteration 338 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:10:48,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:48,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:48,262][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:48,295][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:48,296][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:48,296][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:49,047][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:49,343][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:49,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:49,993][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:50,318][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:50,644][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:50,970][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:51,297][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:51,625][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:51,953][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:52,278][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:52,603][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:52,929][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:53,258][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:53,583][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:53,908][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:54,232][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:54,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:54,881][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:55,206][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:55,530][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:56,181][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:56,505][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:56,830][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:10:57,156][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:10:57,483][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:10:57,813][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:10:58,141][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:10:58,468][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:10:58,794][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:10:59,120][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:10:59,446][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:00,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:00,957][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:00,960][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:00,963][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:01,872][__main__][INFO] - Iteration 339 took 23s (40.68% Gen, 55.51% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 21s. Estimated total time: 19h 53m 34s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 47s, 500 more iterations: 3h 18m 55s.
[2025-11-13 10:11:01,874][__main__][INFO] - Starting iteration 339.
[2025-11-13 10:11:01,877][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:11:01,877][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:11:11,596][__main__][INFO] - Number of regex retries in iteration 339: 0
[2025-11-13 10:11:11,596][__main__][INFO] - agents played in iteration 339 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:11:12,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:12,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:12,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:12,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:12,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:11:12,177][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:11:12,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:11:13,234][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:11:13,561][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:11:13,890][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:11:14,217][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:11:14,548][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:11:14,873][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:11:15,197][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:11:15,524][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:11:15,851][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:11:16,177][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:11:16,503][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:11:16,831][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:11:17,156][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:11:17,482][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:11:17,806][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:11:18,136][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:11:18,462][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:11:18,789][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:11:19,114][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:11:19,441][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:11:19,768][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:11:20,093][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:11:20,420][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:11:20,746][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:11:21,072][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:11:21,399][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:11:21,725][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:11:22,053][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:11:22,383][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:11:22,711][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:11:23,039][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:11:23,363][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:24,095][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:24,794][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:24,795][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:24,797][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:25,708][__main__][INFO] - Iteration 340 took 23s (40.78% Gen, 55.39% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 43m 59s. Estimated total time: 19h 51m 36s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 36s.
[2025-11-13 10:11:25,710][__main__][INFO] - Starting iteration 340.
[2025-11-13 10:11:25,713][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:11:25,713][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:11:35,384][__main__][INFO] - Number of regex retries in iteration 340: 0
[2025-11-13 10:11:35,384][__main__][INFO] - agents played in iteration 340 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:11:35,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:35,910][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:35,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:35,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:35,977][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:11:35,977][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:11:36,730][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:11:37,026][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:11:37,351][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:11:37,681][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:11:38,004][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:11:38,329][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:11:38,655][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:11:38,980][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:11:39,305][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:11:39,630][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:11:39,956][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:11:40,282][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:11:40,608][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:11:40,939][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:11:41,265][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:11:41,592][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:11:41,918][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:11:42,242][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:11:42,566][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:11:42,892][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:11:43,217][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:11:43,542][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:11:43,868][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:11:44,194][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:11:44,526][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:11:44,853][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:11:45,182][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:11:45,509][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:11:45,848][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:11:46,173][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:11:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:11:46,826][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:11:47,156][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:47,889][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:48,625][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:48,627][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:48,630][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:50,614][__main__][INFO] - Iteration 341 took 24s (38.83% Gen, 53.19% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 37m 6s. Estimated total time: 20h 45m 8s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 30s, 500 more iterations: 3h 27m 31s.
[2025-11-13 10:11:50,616][__main__][INFO] - Starting iteration 341.
[2025-11-13 10:11:50,619][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:11:50,620][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:11:59,896][__main__][INFO] - Number of regex retries in iteration 341: 0
[2025-11-13 10:11:59,897][__main__][INFO] - agents played in iteration 341 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:12:00,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:00,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:00,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:00,481][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:00,481][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:00,482][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:01,241][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:01,539][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:01,865][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:02,195][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:02,522][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:02,854][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:03,182][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:03,509][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:03,835][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:04,161][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:04,488][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:04,820][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:05,147][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:05,477][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:05,804][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:06,129][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:06,454][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:06,781][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:07,106][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:07,436][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:07,768][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:08,088][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:08,416][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:08,744][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:09,070][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:12:09,397][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:12:09,721][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:12:10,048][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:12:10,379][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:12:10,702][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:12:11,028][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:12:11,353][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:12:11,678][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:12:12,449][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:12:13,143][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:12:13,145][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:12:13,147][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:12:14,128][__main__][INFO] - Iteration 342 took 23s (39.46% Gen, 56.36% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 27m 3s. Estimated total time: 19h 35m 28s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 54s.
[2025-11-13 10:12:14,130][__main__][INFO] - Starting iteration 342.
[2025-11-13 10:12:14,133][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:12:14,133][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:12:23,696][__main__][INFO] - Number of regex retries in iteration 342: 0
[2025-11-13 10:12:23,697][__main__][INFO] - agents played in iteration 342 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:12:24,192][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:24,225][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:24,258][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:24,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:24,292][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:24,293][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:25,043][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:25,346][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:25,672][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:26,001][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:26,327][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:26,664][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:26,994][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:27,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:27,655][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:27,981][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:28,305][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:28,632][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:28,961][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:29,287][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:29,612][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:29,938][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:30,263][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:30,587][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:30,911][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:31,236][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:31,561][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:31,887][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:32,213][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:32,540][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:32,866][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:12:33,191][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:12:33,516][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:12:33,843][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:12:34,170][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:12:34,496][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:12:34,821][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:12:35,147][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:12:35,478][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:12:36,221][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:12:36,928][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:12:36,930][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:12:36,932][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:12:37,878][__main__][INFO] - Iteration 343 took 23s (40.28% Gen, 55.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 38m 29s. Estimated total time: 19h 47m 18s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 34s, 500 more iterations: 3h 17m 53s.
[2025-11-13 10:12:37,881][__main__][INFO] - Starting iteration 343.
[2025-11-13 10:12:37,884][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:12:37,885][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:12:46,410][__main__][INFO] - Number of regex retries in iteration 343: 0
[2025-11-13 10:12:46,411][__main__][INFO] - agents played in iteration 343 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:12:46,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:46,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:46,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:46,996][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:46,997][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:46,997][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:47,765][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:48,063][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:48,389][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:48,715][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:49,042][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:49,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:49,692][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:50,023][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:50,351][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:50,678][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:51,003][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:51,328][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:51,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:51,986][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:52,310][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:52,636][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:53,289][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:53,617][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:53,940][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:54,265][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:54,592][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:54,923][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:55,252][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:55,579][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:12:55,910][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:12:56,237][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:12:56,562][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:12:56,888][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:12:57,217][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:12:57,545][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:12:57,871][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:12:58,199][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:12:58,940][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:12:59,649][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:12:59,651][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:12:59,652][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:00,576][__main__][INFO] - Iteration 344 took 22s (37.57% Gen, 58.35% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 45m 28s. Estimated total time: 18h 54m 39s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 49s, 500 more iterations: 3h 9m 6s.
[2025-11-13 10:13:00,579][__main__][INFO] - Starting iteration 344.
[2025-11-13 10:13:00,582][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:00,582][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:13:09,324][__main__][INFO] - Number of regex retries in iteration 344: 0
[2025-11-13 10:13:09,325][__main__][INFO] - agents played in iteration 344 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:13:09,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:09,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:09,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:09,904][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:09,905][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:13:09,905][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:13:10,657][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:13:10,954][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:13:11,281][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:13:11,606][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:13:11,931][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:13:12,257][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:13:12,582][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:13:12,907][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:13:13,232][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:13:13,557][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:13:13,883][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:13:14,214][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:13:14,540][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:13:14,865][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:13:15,194][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:13:15,520][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:13:15,846][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:13:16,173][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:13:16,501][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:13:16,826][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:13:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:13:17,481][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:13:17,810][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:13:18,139][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:13:18,464][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:13:18,788][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:13:19,114][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:13:19,440][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:13:19,767][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:13:20,092][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:13:20,418][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:13:20,743][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:13:21,068][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:13:21,812][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:13:22,503][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:13:22,505][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:13:22,507][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:23,524][__main__][INFO] - Iteration 345 took 22s (38.10% Gen, 57.46% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 57m 36s. Estimated total time: 19h 7m 11s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 14s, 500 more iterations: 3h 11m 11s.
[2025-11-13 10:13:23,528][__main__][INFO] - Starting iteration 345.
[2025-11-13 10:13:23,531][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:23,532][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:13:32,930][__main__][INFO] - Number of regex retries in iteration 345: 0
[2025-11-13 10:13:32,930][__main__][INFO] - agents played in iteration 345 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:13:33,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:33,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:33,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:33,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:33,522][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:13:33,522][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:13:34,290][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:13:34,587][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:13:34,913][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:13:35,240][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:13:35,574][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:13:35,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:13:36,224][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:13:36,549][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:13:36,885][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:13:37,210][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:13:37,537][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:13:37,862][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:13:38,195][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:13:38,520][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:13:38,847][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:13:39,173][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:13:39,501][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:13:39,828][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:13:40,152][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:13:40,484][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:13:40,805][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:13:41,131][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:13:41,455][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:13:41,784][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:13:42,104][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:13:42,429][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:13:42,754][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:13:43,082][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:13:43,404][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:13:43,732][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:13:44,059][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:13:44,391][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:13:44,711][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:13:45,444][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:13:46,153][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:13:46,154][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:13:46,156][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:47,137][__main__][INFO] - Iteration 346 took 23s (39.81% Gen, 56.02% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 30m 23s. Estimated total time: 19h 40m 22s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 20s, 500 more iterations: 3h 16m 43s.
[2025-11-13 10:13:47,140][__main__][INFO] - Starting iteration 346.
[2025-11-13 10:13:47,145][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:47,145][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:13:55,874][__main__][INFO] - Number of regex retries in iteration 346: 0
[2025-11-13 10:13:55,875][__main__][INFO] - agents played in iteration 346 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:13:56,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:56,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:56,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:56,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:56,469][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:13:56,470][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:13:57,223][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:13:57,521][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:13:57,846][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:13:58,177][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:13:58,503][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:13:58,829][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:13:59,155][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:13:59,480][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:13:59,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:00,132][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:00,459][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:00,784][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:01,111][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:01,437][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:01,763][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:02,089][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:02,415][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:02,745][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:03,407][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:03,733][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:04,059][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:04,383][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:04,709][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:05,034][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:05,360][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:05,692][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:06,017][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:06,346][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:06,673][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:07,324][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:07,649][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:14:08,402][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:14:09,102][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:14:09,104][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:14:09,105][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:14:10,120][__main__][INFO] - Iteration 347 took 22s (37.99% Gen, 57.58% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 58m 31s. Estimated total time: 19h 8m 52s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 17s, 500 more iterations: 3h 11m 28s.
[2025-11-13 10:14:10,122][__main__][INFO] - Starting iteration 347.
[2025-11-13 10:14:10,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:14:10,125][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:14:16,664][mllm.models.large_language_model_local][WARNING] - Response %A> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 10:14:19,553][__main__][INFO] - Number of regex retries in iteration 347: 1
[2025-11-13 10:14:19,554][__main__][INFO] - agents played in iteration 347 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:14:20,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:20,079][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:20,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:20,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:20,147][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:14:20,148][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:14:20,905][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:14:21,202][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:14:21,532][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:14:21,858][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:14:22,185][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:14:22,519][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:14:22,840][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:14:23,165][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:14:23,493][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:23,819][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:24,144][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:24,469][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:24,798][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:25,124][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:25,451][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:25,778][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:26,104][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:26,432][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:26,758][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:27,084][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:27,414][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:27,742][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:28,069][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:28,399][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:28,730][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:29,056][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:29,391][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:29,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:30,043][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:30,368][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:30,704][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:31,354][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:14:32,088][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:14:32,783][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:14:32,786][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:14:32,787][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:14:33,716][__main__][INFO] - Iteration 348 took 23s (39.96% Gen, 56.09% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 28m 51s. Estimated total time: 19h 39m 36s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 36s.
[2025-11-13 10:14:33,718][__main__][INFO] - Starting iteration 348.
[2025-11-13 10:14:33,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:14:33,721][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:14:42,388][__main__][INFO] - Number of regex retries in iteration 348: 0
[2025-11-13 10:14:42,389][__main__][INFO] - agents played in iteration 348 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:14:42,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:42,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:42,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:42,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:42,980][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:14:42,981][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:14:43,750][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:14:44,048][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:14:44,375][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:14:44,701][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:14:45,031][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:14:45,357][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:14:45,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:14:46,013][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:14:46,340][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:46,672][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:47,000][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:47,328][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:47,655][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:47,981][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:48,309][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:48,634][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:48,959][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:49,289][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:49,615][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:49,941][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:50,266][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:50,597][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:50,923][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:51,249][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:51,574][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:51,903][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:52,235][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:52,560][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:52,887][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:53,214][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:53,542][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:53,867][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:54,193][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:14:54,958][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:14:55,680][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:14:55,682][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:14:55,684][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:14:56,591][__main__][INFO] - Iteration 349 took 22s (37.90% Gen, 58.13% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 52m 25s. Estimated total time: 19h 3m 33s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 7s, 500 more iterations: 3h 10m 35s.
[2025-11-13 10:14:56,593][__main__][INFO] - Starting iteration 349.
[2025-11-13 10:14:56,596][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:14:56,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:05,657][__main__][INFO] - Number of regex retries in iteration 349: 0
[2025-11-13 10:15:05,657][__main__][INFO] - agents played in iteration 349 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:15:06,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:06,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:06,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:06,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:06,247][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:06,248][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:15:07,016][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:15:07,312][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:15:07,639][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:15:07,967][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:15:08,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:15:08,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:15:08,945][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:15:09,269][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:15:09,597][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:15:09,922][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:15:10,247][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:15:10,579][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:15:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:15:11,239][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:15:11,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:15:11,892][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:15:12,219][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:15:12,546][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:15:12,873][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:15:13,197][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:15:13,523][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:15:13,848][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:15:14,173][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:15:14,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:15:14,824][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:15:15,154][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:15:15,486][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:15:15,816][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:15:16,147][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:15:16,474][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:15:16,806][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:15:17,132][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:15:17,458][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:15:18,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:15:18,895][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:15:18,897][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:15:18,898][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:15:19,830][__main__][INFO] - Iteration 350 took 23s (39.00% Gen, 56.98% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 16s. Estimated total time: 19h 21m 47s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 37s.
[2025-11-13 10:15:19,832][__main__][INFO] - Starting iteration 350.
[2025-11-13 10:15:19,835][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:15:19,836][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:28,628][__main__][INFO] - Number of regex retries in iteration 350: 0
[2025-11-13 10:15:28,629][__main__][INFO] - agents played in iteration 350 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:15:29,117][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:29,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:29,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:29,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:29,218][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:29,219][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:15:29,984][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:15:30,280][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:15:30,606][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:15:30,932][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:15:31,258][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:15:31,584][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:15:31,909][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:15:32,235][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:15:32,560][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:15:32,888][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:15:33,218][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:15:33,546][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:15:33,874][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:15:34,201][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:15:34,524][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:15:34,850][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:15:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:15:35,516][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:15:35,843][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:15:36,169][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:15:36,496][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:15:36,822][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:15:37,149][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:15:37,476][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:15:37,805][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:15:38,134][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:15:38,460][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:15:38,787][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:15:39,117][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:15:39,439][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:15:39,764][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:15:40,092][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:15:40,420][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:15:41,172][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:15:41,891][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:15:41,893][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:15:41,894][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:15:43,782][__main__][INFO] - Iteration 351 took 23s (36.72% Gen, 55.39% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 45m 28s. Estimated total time: 19h 57m 23s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 54s, 500 more iterations: 3h 19m 33s.
[2025-11-13 10:15:43,784][__main__][INFO] - Starting iteration 351.
[2025-11-13 10:15:43,787][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:15:43,787][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:52,912][__main__][INFO] - Number of regex retries in iteration 351: 0
[2025-11-13 10:15:52,913][__main__][INFO] - agents played in iteration 351 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:15:53,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:53,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:53,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:53,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:53,497][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:53,498][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:15:54,256][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:15:54,553][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:15:54,879][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:15:55,204][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:15:55,529][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:15:55,855][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:15:56,182][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:15:56,507][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:15:56,834][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:15:57,158][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:15:57,484][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:15:57,813][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:15:58,142][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:15:58,469][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:15:58,793][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:15:59,126][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:15:59,454][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:15:59,780][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:00,106][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:00,436][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:00,765][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:01,091][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:01,421][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:01,743][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:02,069][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:02,395][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:02,721][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:03,049][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:03,707][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:04,040][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:04,361][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:04,687][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:05,431][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:06,144][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:06,147][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:06,149][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:16:07,069][__main__][INFO] - Iteration 352 took 23s (39.19% Gen, 56.85% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 11m 51s. Estimated total time: 19h 24m 9s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 1s.
[2025-11-13 10:16:07,071][__main__][INFO] - Starting iteration 352.
[2025-11-13 10:16:07,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:16:07,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:16:15,798][__main__][INFO] - Number of regex retries in iteration 352: 0
[2025-11-13 10:16:15,798][__main__][INFO] - agents played in iteration 352 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:16:16,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:16,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:16,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:16,404][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:16,405][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:16:16,405][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:16:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:16:17,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:16:17,793][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:16:18,118][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:16:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:16:18,769][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:16:19,094][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:16:19,421][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:16:19,748][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:16:20,074][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:16:20,400][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:16:20,726][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:16:21,051][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:16:21,380][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:16:21,705][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:16:22,036][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:16:22,364][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:16:22,690][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:23,016][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:23,343][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:23,669][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:23,995][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:24,326][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:24,657][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:24,983][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:25,309][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:25,639][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:25,967][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:26,295][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:26,623][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:26,949][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:27,275][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:27,601][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:28,347][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:29,056][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:29,058][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:29,059][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:16:30,024][__main__][INFO] - Iteration 353 took 22s (38.01% Gen, 57.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 54m 50s. Estimated total time: 19h 7m 31s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 15s.
[2025-11-13 10:16:30,026][__main__][INFO] - Starting iteration 353.
[2025-11-13 10:16:30,029][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:16:30,029][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:16:39,497][__main__][INFO] - Number of regex retries in iteration 353: 0
[2025-11-13 10:16:39,498][__main__][INFO] - agents played in iteration 353 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:16:39,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:40,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:40,060][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:40,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:40,094][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:16:40,095][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:16:40,851][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:16:41,147][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:16:41,474][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:16:41,799][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:16:42,124][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:16:42,453][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:16:42,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:16:43,105][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:16:43,431][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:16:43,759][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:16:44,086][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:16:44,419][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:16:44,746][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:16:45,073][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:16:45,404][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:16:45,731][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:16:46,057][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:16:46,383][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:46,709][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:47,037][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:47,362][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:47,688][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:48,015][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:48,340][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:48,668][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:48,994][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:49,321][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:49,649][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:49,976][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:50,304][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:50,955][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:51,281][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:52,032][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:52,745][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:52,747][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:52,749][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:16:53,805][__main__][INFO] - Iteration 354 took 23s (39.82% Gen, 55.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 35m 46s. Estimated total time: 19h 48m 51s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 37s, 500 more iterations: 3h 18m 8s.
[2025-11-13 10:16:53,807][__main__][INFO] - Starting iteration 354.
[2025-11-13 10:16:53,810][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:16:53,810][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:02,854][__main__][INFO] - Number of regex retries in iteration 354: 0
[2025-11-13 10:17:02,855][__main__][INFO] - agents played in iteration 354 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:17:03,361][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:03,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:03,442][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:03,481][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:03,481][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:03,482][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:04,250][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:04,547][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:04,874][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:17:05,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:17:05,524][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:17:05,851][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:17:06,176][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:17:06,502][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:17:06,828][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:17:07,154][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:17:07,481][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:17:07,806][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:17:08,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:17:08,458][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:17:08,785][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:17:09,113][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:17:09,439][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:17:09,767][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:17:10,098][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:17:10,429][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:17:10,756][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:17:11,087][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:17:11,420][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:17:11,747][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:17:12,075][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:17:12,403][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:17:12,730][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:17:13,055][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:17:13,381][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:17:13,706][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:17:14,043][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:17:14,362][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:17:14,689][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:17:15,450][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:17:16,147][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:17:16,149][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:17:16,151][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:17:17,072][__main__][INFO] - Iteration 355 took 23s (38.88% Gen, 57.15% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 9m 41s. Estimated total time: 19h 23m 9s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 46s, 500 more iterations: 3h 13m 51s.
[2025-11-13 10:17:17,075][__main__][INFO] - Starting iteration 355.
[2025-11-13 10:17:17,078][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:17:17,079][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:25,999][__main__][INFO] - Number of regex retries in iteration 355: 0
[2025-11-13 10:17:25,999][__main__][INFO] - agents played in iteration 355 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:17:26,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:26,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:26,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:26,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:26,598][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:26,598][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:27,360][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:27,657][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:27,983][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:17:28,311][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:17:28,637][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:17:28,963][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:17:29,290][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:17:29,615][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:17:29,941][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:17:30,267][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:17:30,592][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:17:30,919][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:17:31,246][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:17:31,572][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:17:31,897][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:17:32,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:17:32,553][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:17:32,879][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:17:33,205][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:17:33,531][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:17:33,859][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:17:34,186][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:17:34,514][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:17:34,840][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:17:35,166][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:17:35,495][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:17:35,823][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:17:36,149][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:17:36,476][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:17:36,805][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:17:37,132][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:17:37,458][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:17:37,784][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:17:38,523][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:17:39,229][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:17:39,231][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:17:39,234][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:17:40,153][__main__][INFO] - Iteration 356 took 23s (38.66% Gen, 57.35% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 59m 55s. Estimated total time: 19h 13m 46s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 27s, 500 more iterations: 3h 12m 17s.
[2025-11-13 10:17:40,156][__main__][INFO] - Starting iteration 356.
[2025-11-13 10:17:40,160][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:17:40,161][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:49,064][__main__][INFO] - Number of regex retries in iteration 356: 0
[2025-11-13 10:17:49,065][__main__][INFO] - agents played in iteration 356 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:17:49,561][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:49,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:49,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:49,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:49,662][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:49,662][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:50,433][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:50,730][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:51,056][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:17:51,381][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:17:51,707][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:17:52,032][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:17:52,358][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:17:52,683][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:17:53,009][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:17:53,336][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:17:53,662][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:17:53,988][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:17:54,313][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:17:54,641][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:17:54,966][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:17:55,291][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:17:55,617][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:17:55,947][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:17:56,270][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:17:56,596][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:17:56,921][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:17:57,252][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:17:57,577][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:17:57,903][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:17:58,229][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:17:58,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:17:58,884][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:17:59,212][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:17:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:17:59,873][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:00,195][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:00,521][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:00,847][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:01,588][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:02,286][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:02,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:02,289][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:03,245][__main__][INFO] - Iteration 357 took 23s (38.57% Gen, 57.28% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 0m 4s. Estimated total time: 19h 14m 18s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 23s.
[2025-11-13 10:18:03,247][__main__][INFO] - Starting iteration 357.
[2025-11-13 10:18:03,250][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:18:03,251][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:18:12,730][__main__][INFO] - Number of regex retries in iteration 357: 0
[2025-11-13 10:18:12,730][__main__][INFO] - agents played in iteration 357 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:18:13,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:13,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:13,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:13,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:13,329][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:18:13,330][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:18:14,094][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:18:14,394][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:18:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:18:15,045][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:18:15,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:18:15,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:18:16,024][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:18:16,350][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:18:16,675][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:18:17,001][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:18:17,326][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:18:17,653][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:18:17,979][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:18:18,307][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:18:18,638][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:18:18,963][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:18:19,292][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:18:19,624][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:18:19,951][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:18:20,281][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:18:20,610][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:18:20,945][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:18:21,270][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:18:21,599][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:18:21,924][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:18:22,255][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:18:22,581][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:18:22,908][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:18:23,242][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:18:23,570][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:23,901][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:24,229][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:24,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:25,312][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:26,033][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:26,035][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:26,036][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:27,003][__main__][INFO] - Iteration 358 took 23s (39.90% Gen, 56.02% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 33m 4s. Estimated total time: 19h 47m 42s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 57s.
[2025-11-13 10:18:27,005][__main__][INFO] - Starting iteration 358.
[2025-11-13 10:18:27,008][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:18:27,009][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:18:36,367][__main__][INFO] - Number of regex retries in iteration 358: 0
[2025-11-13 10:18:36,367][__main__][INFO] - agents played in iteration 358 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:18:36,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:36,889][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:36,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:36,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:36,957][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:18:36,958][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:18:37,718][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:18:38,014][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:18:38,340][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:18:38,666][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:18:38,991][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:18:39,317][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:18:39,642][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:18:39,968][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:18:40,294][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:18:40,620][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:18:40,945][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:18:41,272][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:18:41,599][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:18:41,931][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:18:42,258][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:18:42,583][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:18:42,910][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:18:43,242][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:18:43,568][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:18:43,895][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:18:44,225][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:18:44,560][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:18:44,886][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:18:45,211][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:18:45,539][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:18:45,869][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:18:46,196][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:18:46,521][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:18:46,852][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:18:47,183][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:47,511][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:47,840][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:48,168][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:48,910][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:49,627][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:49,629][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:49,631][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:50,563][__main__][INFO] - Iteration 359 took 23s (39.73% Gen, 56.31% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 22m 43s. Estimated total time: 19h 37m 45s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 17s.
[2025-11-13 10:18:50,565][__main__][INFO] - Starting iteration 359.
[2025-11-13 10:18:50,567][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:18:50,568][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:18:59,944][__main__][INFO] - Number of regex retries in iteration 359: 0
[2025-11-13 10:18:59,945][__main__][INFO] - agents played in iteration 359 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:19:00,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:00,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:00,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:00,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:00,551][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:00,552][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:01,313][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:01,611][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:01,945][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:02,272][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:02,598][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:02,923][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:19:03,249][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:19:03,574][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:19:03,899][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:19:04,224][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:19:04,563][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:19:04,889][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:19:05,215][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:19:05,540][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:19:05,871][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:19:06,196][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:19:06,522][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:19:06,847][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:19:07,175][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:19:07,500][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:19:07,827][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:19:08,152][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:19:08,482][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:19:08,808][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:19:09,134][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:19:09,466][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:19:09,788][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:19:10,116][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:19:10,443][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:19:10,777][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:19:11,101][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:19:11,430][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:19:11,758][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:19:12,508][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:19:13,238][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:19:13,241][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:19:13,243][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:19:14,179][__main__][INFO] - Iteration 360 took 23s (39.71% Gen, 56.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 25m 11s. Estimated total time: 19h 40m 36s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 46s.
[2025-11-13 10:19:14,181][__main__][INFO] - Starting iteration 360.
[2025-11-13 10:19:14,184][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:19:14,184][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:19:23,028][__main__][INFO] - Number of regex retries in iteration 360: 0
[2025-11-13 10:19:23,029][__main__][INFO] - agents played in iteration 360 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:19:23,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:23,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:23,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:24,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:24,003][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:24,004][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:24,774][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:25,071][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:25,398][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:25,723][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:26,050][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:26,377][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:19:26,702][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:19:27,027][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:19:27,354][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:19:27,680][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:19:28,008][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:19:28,334][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:19:28,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:19:28,985][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:19:29,314][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:19:29,643][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:19:29,971][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:19:30,296][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:19:30,621][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:19:30,946][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:19:31,272][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:19:31,604][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:19:31,933][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:19:32,260][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:19:32,588][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:19:32,915][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:19:33,242][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:19:33,569][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:19:33,895][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:19:34,221][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:19:34,550][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:19:34,881][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:19:35,208][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:19:35,995][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:19:36,759][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:19:36,904][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:19:36,906][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:19:38,845][__main__][INFO] - Iteration 361 took 24s (35.86% Gen, 56.27% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 17m 17s. Estimated total time: 20h 33m 7s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 6s, 500 more iterations: 3h 25m 31s.
[2025-11-13 10:19:38,847][__main__][INFO] - Starting iteration 361.
[2025-11-13 10:19:38,850][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:19:38,850][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:19:48,052][__main__][INFO] - Number of regex retries in iteration 361: 0
[2025-11-13 10:19:48,053][__main__][INFO] - agents played in iteration 361 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:19:48,562][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:48,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:48,629][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:48,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:48,664][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:48,665][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:49,460][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:49,758][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:50,094][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:50,412][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:50,739][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:51,065][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:19:51,390][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:19:51,716][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:19:52,043][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:19:52,371][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:19:52,697][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:19:53,028][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:19:53,354][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:19:53,680][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:19:54,006][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:19:54,332][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:19:54,657][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:19:54,982][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:19:55,309][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:19:55,638][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:19:55,963][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:19:56,288][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:19:56,616][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:19:56,943][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:19:57,272][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:19:57,600][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:19:57,925][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:19:58,256][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:19:58,580][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:19:58,905][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:19:59,231][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:19:59,561][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:19:59,887][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:00,673][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:01,429][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:01,430][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:01,432][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:02,422][__main__][INFO] - Iteration 362 took 23s (39.04% Gen, 56.76% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 22m 24s. Estimated total time: 19h 38m 38s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 26s.
[2025-11-13 10:20:02,424][__main__][INFO] - Starting iteration 362.
[2025-11-13 10:20:02,427][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:02,427][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:20:12,005][__main__][INFO] - Number of regex retries in iteration 362: 0
[2025-11-13 10:20:12,006][__main__][INFO] - agents played in iteration 362 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:20:12,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:12,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:12,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:12,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:12,607][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:20:12,607][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:20:13,380][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:20:13,676][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:20:14,004][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:20:14,326][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:20:14,651][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:20:14,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:20:15,304][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:20:15,628][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:20:15,954][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:20:16,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:20:16,604][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:20:16,930][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:20:17,254][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:20:17,581][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:20:17,905][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:20:18,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:20:18,558][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:20:18,884][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:20:19,209][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:20:19,533][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:20:19,862][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:20:20,191][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:20:20,517][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:20:20,843][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:20:21,169][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:20:21,494][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:20:21,819][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:20:22,144][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:20:22,469][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:20:22,793][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:20:23,123][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:20:23,451][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:20:23,778][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:24,527][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:25,249][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:25,250][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:25,252][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:26,231][__main__][INFO] - Iteration 363 took 23s (40.24% Gen, 55.65% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 33m 37s. Estimated total time: 19h 50m 14s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 40s, 500 more iterations: 3h 18m 22s.
[2025-11-13 10:20:26,233][__main__][INFO] - Starting iteration 363.
[2025-11-13 10:20:26,236][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:26,237][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:20:35,787][__main__][INFO] - Number of regex retries in iteration 363: 0
[2025-11-13 10:20:35,788][__main__][INFO] - agents played in iteration 363 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:20:36,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:36,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:36,358][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:36,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:36,392][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:20:36,393][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:20:37,166][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:20:37,464][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:20:37,794][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:20:38,120][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:20:38,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:20:38,772][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:20:39,097][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:20:39,423][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:20:39,749][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:20:40,074][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:20:40,402][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:20:40,730][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:20:41,056][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:20:41,383][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:20:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:20:42,034][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:20:42,360][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:20:42,684][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:20:43,011][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:20:43,338][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:20:43,663][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:20:43,992][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:20:44,316][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:20:44,642][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:20:44,966][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:20:45,291][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:20:45,619][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:20:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:20:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:20:46,595][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:20:46,923][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:20:47,254][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:20:47,580][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:48,328][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:49,054][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:49,056][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:49,057][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:50,070][__main__][INFO] - Iteration 364 took 23s (40.07% Gen, 55.67% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 34m 40s. Estimated total time: 19h 51m 42s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 37s.
[2025-11-13 10:20:50,071][__main__][INFO] - Starting iteration 364.
[2025-11-13 10:20:50,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:50,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:20:59,167][__main__][INFO] - Number of regex retries in iteration 364: 0
[2025-11-13 10:20:59,167][__main__][INFO] - agents played in iteration 364 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:20:59,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:59,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:59,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:59,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:59,768][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:20:59,768][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:00,567][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:00,865][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:01,197][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:01,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:01,843][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:02,170][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:02,496][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:02,822][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:03,148][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:03,474][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:03,801][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:04,128][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:04,453][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:04,780][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:05,107][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:05,433][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:05,760][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:06,088][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:21:06,416][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:21:06,743][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:21:07,069][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:21:07,396][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:21:07,723][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:21:08,048][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:21:08,373][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:21:08,700][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:21:09,024][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:21:09,348][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:21:09,674][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:21:10,001][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:21:10,329][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:21:10,654][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:21:10,981][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:21:11,725][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:21:12,476][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:21:12,477][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:21:12,480][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:21:13,425][__main__][INFO] - Iteration 365 took 23s (38.93% Gen, 57.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 10s. Estimated total time: 19h 27m 35s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 35s.
[2025-11-13 10:21:13,427][__main__][INFO] - Starting iteration 365.
[2025-11-13 10:21:13,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:21:13,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:21:22,740][__main__][INFO] - Number of regex retries in iteration 365: 0
[2025-11-13 10:21:22,740][__main__][INFO] - agents played in iteration 365 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:21:23,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:23,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:23,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:23,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:23,338][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:21:23,339][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:24,120][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:24,416][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:24,742][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:25,068][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:25,393][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:25,719][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:26,046][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:26,372][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:26,697][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:27,023][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:27,348][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:27,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:28,004][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:28,329][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:28,655][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:28,986][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:29,313][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:29,643][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:21:29,971][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:21:30,304][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:21:30,630][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:21:30,957][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:21:31,282][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:21:31,609][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:21:31,937][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:21:32,262][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:21:32,593][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:21:32,918][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:21:33,250][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:21:33,577][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:21:33,915][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:21:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:21:34,562][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:21:35,299][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:21:36,024][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:21:36,028][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:21:36,029][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:21:36,992][__main__][INFO] - Iteration 366 took 23s (39.51% Gen, 56.40% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 20m 18s. Estimated total time: 19h 38m 6s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 21s.
[2025-11-13 10:21:36,994][__main__][INFO] - Starting iteration 366.
[2025-11-13 10:21:36,997][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:21:36,997][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:21:46,527][__main__][INFO] - Number of regex retries in iteration 366: 0
[2025-11-13 10:21:46,527][__main__][INFO] - agents played in iteration 366 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:21:47,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:47,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:47,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:47,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:47,131][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:21:47,131][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:47,900][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:48,522][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:48,848][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:49,174][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:49,500][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:49,826][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:50,151][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:50,478][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:50,803][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:51,129][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:51,456][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:51,781][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:52,107][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:52,433][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:52,760][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:53,087][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:53,413][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:21:53,744][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:21:54,071][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:21:54,396][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:21:54,725][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:21:55,049][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:21:55,375][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:21:55,701][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:21:56,028][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:21:56,363][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:21:56,688][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:21:57,014][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:21:57,340][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:21:57,667][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:21:57,993][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:21:58,318][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:21:59,054][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:21:59,780][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:21:59,781][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:21:59,783][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:00,740][__main__][INFO] - Iteration 367 took 23s (40.13% Gen, 55.83% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 28m 59s. Estimated total time: 19h 47m 11s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 34s, 500 more iterations: 3h 17m 51s.
[2025-11-13 10:22:00,742][__main__][INFO] - Starting iteration 367.
[2025-11-13 10:22:00,745][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:22:00,746][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:22:10,300][__main__][INFO] - Number of regex retries in iteration 367: 0
[2025-11-13 10:22:10,301][__main__][INFO] - agents played in iteration 367 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:22:10,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:10,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:10,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:10,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:10,907][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:22:10,908][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:22:11,666][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:22:11,962][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:22:12,288][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:22:12,613][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:22:12,939][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:22:13,266][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:22:13,605][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:22:13,930][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:22:14,256][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:22:14,584][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:22:14,914][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:22:15,240][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:22:15,565][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:22:15,891][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:22:16,218][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:22:16,547][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:22:16,874][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:22:17,209][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:22:17,530][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:22:17,856][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:22:18,180][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:22:18,510][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:22:18,832][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:22:19,157][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:22:19,485][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:22:19,812][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:22:20,138][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:22:20,465][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:22:20,794][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:22:21,127][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:22:21,446][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:22:21,771][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:22:22,096][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:22:22,819][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:22:23,555][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:22:23,559][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:22:23,563][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:24,527][__main__][INFO] - Iteration 368 took 23s (40.18% Gen, 55.76% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 30m 32s. Estimated total time: 19h 49m 8s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 38s, 500 more iterations: 3h 18m 11s.
[2025-11-13 10:22:24,530][__main__][INFO] - Starting iteration 368.
[2025-11-13 10:22:24,533][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:22:24,533][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:22:34,174][__main__][INFO] - Number of regex retries in iteration 368: 0
[2025-11-13 10:22:34,175][__main__][INFO] - agents played in iteration 368 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:22:34,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:34,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:34,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:34,769][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:34,770][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:22:34,771][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:22:35,542][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:22:35,836][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:22:36,162][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:22:36,489][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:22:36,815][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:22:37,142][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:22:37,467][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:22:37,793][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:22:38,124][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:22:38,443][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:22:38,767][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:22:39,094][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:22:39,426][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:22:39,745][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:22:40,070][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:22:40,398][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:22:40,733][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:22:41,054][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:22:41,381][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:22:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:22:42,039][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:22:42,360][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:22:42,684][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:22:43,009][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:22:43,342][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:22:43,663][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:22:43,991][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:22:44,317][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:22:44,646][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:22:44,968][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:22:45,296][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:22:45,623][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:22:45,951][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:22:46,684][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:22:47,428][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:22:47,430][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:22:47,433][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:48,329][__main__][INFO] - Iteration 369 took 23s (40.52% Gen, 55.71% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 30m 51s. Estimated total time: 19h 49m 51s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 39s, 500 more iterations: 3h 18m 18s.
[2025-11-13 10:22:48,331][__main__][INFO] - Starting iteration 369.
[2025-11-13 10:22:48,334][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:22:48,335][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:22:57,959][__main__][INFO] - Number of regex retries in iteration 369: 0
[2025-11-13 10:22:57,960][__main__][INFO] - agents played in iteration 369 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:22:58,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:58,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:58,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:58,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:58,553][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:22:58,554][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:22:59,313][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:22:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:22:59,935][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:00,262][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:00,592][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:00,919][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:01,245][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:01,571][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:01,901][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:02,228][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:02,555][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:02,880][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:03,208][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:03,539][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:03,866][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:04,193][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:04,520][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:04,849][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:05,180][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:05,507][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:05,834][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:06,162][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:06,486][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:06,813][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:07,138][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:07,464][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:23:07,790][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:23:08,115][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:23:08,440][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:23:08,771][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:23:09,097][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:23:09,424][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:23:09,748][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:23:10,499][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:23:11,210][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:23:11,212][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:23:11,214][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:23:12,184][__main__][INFO] - Iteration 370 took 23s (40.35% Gen, 55.57% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 33m 8s. Estimated total time: 19h 52m 31s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 45s, 500 more iterations: 3h 18m 45s.
[2025-11-13 10:23:12,187][__main__][INFO] - Starting iteration 370.
[2025-11-13 10:23:12,190][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:23:12,190][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:23:20,777][__main__][INFO] - Number of regex retries in iteration 370: 0
[2025-11-13 10:23:20,777][__main__][INFO] - agents played in iteration 370 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:23:21,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:21,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:21,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:21,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:21,370][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:23:21,370][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:23:22,152][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:23:22,450][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:23:22,777][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:23,102][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:23,427][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:23,752][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:24,078][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:24,405][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:24,732][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:25,057][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:25,384][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:25,722][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:26,047][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:26,373][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:26,701][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:27,027][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:27,354][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:27,683][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:28,012][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:28,339][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:28,667][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:28,993][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:29,323][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:29,645][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:29,973][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:30,298][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:23:30,631][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:23:30,953][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:23:31,281][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:23:31,605][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:23:31,938][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:23:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:23:32,579][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:23:33,306][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:23:34,047][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:23:34,048][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:23:34,050][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:23:35,983][__main__][INFO] - Iteration 371 took 23s (36.09% Gen, 55.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 29m 54s. Estimated total time: 19h 49m 41s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 39s, 500 more iterations: 3h 18m 16s.
[2025-11-13 10:23:35,984][__main__][INFO] - Starting iteration 371.
[2025-11-13 10:23:35,987][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:23:35,988][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:23:45,243][__main__][INFO] - Number of regex retries in iteration 371: 0
[2025-11-13 10:23:45,243][__main__][INFO] - agents played in iteration 371 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:23:45,733][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:45,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:45,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:45,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:45,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:23:45,839][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:23:46,632][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:23:46,928][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:23:47,256][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:47,582][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:47,910][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:48,240][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:48,566][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:48,893][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:49,221][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:49,548][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:49,875][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:50,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:50,529][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:50,857][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:51,182][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:51,507][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:51,833][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:52,484][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:52,809][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:53,134][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:53,459][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:53,784][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:54,111][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:54,440][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:54,766][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:23:55,091][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:23:55,416][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:23:55,747][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:23:56,068][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:23:56,394][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:23:56,721][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:23:57,046][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:23:57,774][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:23:58,478][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:23:58,479][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:23:58,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:23:59,452][__main__][INFO] - Iteration 372 took 23s (39.44% Gen, 56.42% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 13m 6s. Estimated total time: 19h 33m 17s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 6s, 500 more iterations: 3h 15m 32s.
[2025-11-13 10:23:59,454][__main__][INFO] - Starting iteration 372.
[2025-11-13 10:23:59,457][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:23:59,458][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:24:07,881][__main__][INFO] - Number of regex retries in iteration 372: 0
[2025-11-13 10:24:07,882][__main__][INFO] - agents played in iteration 372 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:24:08,362][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:08,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:08,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:08,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:08,464][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:24:08,464][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:24:09,238][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:24:09,536][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:24:09,863][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:24:10,192][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:24:10,517][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:24:10,843][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:24:11,170][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:24:11,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:24:11,821][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:24:12,146][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:24:12,472][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:24:12,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:24:13,127][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:24:13,452][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:24:13,778][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:24:14,105][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:24:14,438][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:24:14,764][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:24:15,089][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:24:15,417][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:24:15,741][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:24:16,067][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:24:16,392][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:24:16,717][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:24:17,045][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:24:17,371][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:24:17,696][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:24:18,022][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:24:18,348][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:24:18,675][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:24:19,000][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:24:19,329][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:24:19,652][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:24:20,404][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:24:21,164][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:24:21,165][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:24:21,167][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:24:22,064][__main__][INFO] - Iteration 373 took 22s (37.26% Gen, 58.77% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 29m 50s. Estimated total time: 18h 50m 23s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 40s, 500 more iterations: 3h 8m 23s.
[2025-11-13 10:24:22,066][__main__][INFO] - Starting iteration 373.
[2025-11-13 10:24:22,069][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:24:22,070][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:24:31,690][__main__][INFO] - Number of regex retries in iteration 373: 0
[2025-11-13 10:24:31,691][__main__][INFO] - agents played in iteration 373 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:24:32,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:32,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:32,262][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:32,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:32,297][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:24:32,297][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:24:33,059][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:24:33,354][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:24:33,682][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:24:34,008][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:24:34,335][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:24:34,661][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:24:34,990][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:24:35,319][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:24:35,646][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:24:35,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:24:36,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:24:36,625][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:24:36,953][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:24:37,279][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:24:37,606][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:24:37,934][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:24:38,259][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:24:38,585][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:24:38,910][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:24:39,236][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:24:39,566][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:24:39,891][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:24:40,217][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:24:40,541][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:24:40,869][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:24:41,198][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:24:41,522][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:24:41,848][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:24:42,173][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:24:42,498][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:24:42,825][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:24:43,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:24:43,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:24:44,219][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:24:44,981][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:24:44,986][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:24:44,988][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:24:46,888][__main__][INFO] - Iteration 374 took 24s (38.76% Gen, 53.57% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 20m 1s. Estimated total time: 20h 40m 59s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 21s, 500 more iterations: 3h 26m 49s.
[2025-11-13 10:24:46,890][__main__][INFO] - Starting iteration 374.
[2025-11-13 10:24:46,893][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:24:46,893][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:24:55,810][__main__][INFO] - Number of regex retries in iteration 374: 0
[2025-11-13 10:24:55,810][__main__][INFO] - agents played in iteration 374 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:24:56,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:56,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:56,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:56,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:56,400][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:24:56,401][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:24:57,149][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:24:57,453][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:24:57,771][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:24:58,096][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:24:58,421][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:24:58,753][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:24:59,071][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:24:59,396][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:24:59,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:00,055][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:00,375][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:00,701][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:01,025][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:01,350][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:01,676][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:02,004][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:02,332][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:02,658][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:02,984][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:03,309][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:03,633][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:04,284][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:04,609][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:04,935][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:05,264][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:05,593][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:05,921][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:06,247][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:06,575][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:06,903][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:07,229][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:07,558][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:08,328][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:09,041][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:09,044][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:09,046][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:25:10,070][__main__][INFO] - Iteration 375 took 23s (38.47% Gen, 57.10% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 57m 33s. Estimated total time: 19h 18m 54s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 9s.
[2025-11-13 10:25:10,072][__main__][INFO] - Starting iteration 375.
[2025-11-13 10:25:10,075][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:25:10,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:25:19,639][__main__][INFO] - Number of regex retries in iteration 375: 0
[2025-11-13 10:25:19,640][__main__][INFO] - agents played in iteration 375 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:25:20,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:20,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:20,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:20,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:20,237][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:25:20,238][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:25:21,008][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:25:21,307][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:25:21,633][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:25:21,960][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:25:22,287][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:25:22,613][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:25:22,938][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:25:23,264][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:25:23,589][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:23,916][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:24,241][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:24,568][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:24,894][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:25,223][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:25,548][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:25,876][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:26,202][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:26,536][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:26,863][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:27,188][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:27,514][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:27,847][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:28,173][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:28,499][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:28,824][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:29,153][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:29,481][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:29,807][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:30,136][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:30,462][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:30,790][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:31,118][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:31,454][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:32,201][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:32,912][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:32,913][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:32,915][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:25:33,844][__main__][INFO] - Iteration 376 took 23s (40.24% Gen, 55.85% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 26m 44s. Estimated total time: 19h 48m 29s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 36s, 500 more iterations: 3h 18m 4s.
[2025-11-13 10:25:33,846][__main__][INFO] - Starting iteration 376.
[2025-11-13 10:25:33,849][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:25:33,850][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:25:43,316][__main__][INFO] - Number of regex retries in iteration 376: 0
[2025-11-13 10:25:43,317][__main__][INFO] - agents played in iteration 376 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:25:43,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:43,849][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:43,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:43,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:43,917][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:25:43,917][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:25:44,670][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:25:44,967][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:25:45,293][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:25:45,618][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:25:45,943][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:25:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:25:46,597][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:25:46,922][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:25:47,248][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:47,575][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:47,901][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:48,227][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:48,551][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:48,877][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:49,204][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:49,528][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:49,854][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:50,179][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:50,506][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:50,830][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:51,159][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:51,486][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:51,813][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:52,141][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:52,469][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:52,797][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:53,129][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:53,460][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:53,791][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:54,119][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:54,451][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:54,777][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:55,103][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:55,846][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:56,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:56,548][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:56,550][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:25:57,576][__main__][INFO] - Iteration 377 took 23s (39.90% Gen, 55.77% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 24m 16s. Estimated total time: 19h 46m 25s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 32s, 500 more iterations: 3h 17m 44s.
[2025-11-13 10:25:57,578][__main__][INFO] - Starting iteration 377.
[2025-11-13 10:25:57,581][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:25:57,581][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:07,421][__main__][INFO] - Number of regex retries in iteration 377: 0
[2025-11-13 10:26:07,421][__main__][INFO] - agents played in iteration 377 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:26:07,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:07,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:07,982][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:08,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:08,015][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:08,016][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:08,789][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:09,084][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:09,411][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:26:09,738][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:26:10,065][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:26:10,390][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:26:10,716][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:26:11,042][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:26:11,370][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:26:11,696][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:26:12,022][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:26:12,349][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:26:12,676][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:26:13,000][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:26:13,326][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:26:13,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:26:13,981][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:26:14,309][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:26:14,634][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:26:14,960][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:26:15,288][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:26:15,615][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:26:15,941][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:26:16,266][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:26:16,591][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:26:16,916][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:26:17,243][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:26:17,569][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:26:17,896][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:26:18,222][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:26:18,549][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:26:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:26:19,204][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:26:19,936][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:26:20,651][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:26:20,653][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:26:20,655][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:26:21,570][__main__][INFO] - Iteration 378 took 23s (41.01% Gen, 55.16% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 36m 58s. Estimated total time: 19h 59m 31s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 59s, 500 more iterations: 3h 19m 55s.
[2025-11-13 10:26:21,573][__main__][INFO] - Starting iteration 378.
[2025-11-13 10:26:21,576][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:26:21,576][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:30,633][__main__][INFO] - Number of regex retries in iteration 378: 0
[2025-11-13 10:26:30,634][__main__][INFO] - agents played in iteration 378 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:26:31,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:31,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:31,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:31,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:31,221][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:31,221][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:31,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:32,293][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:32,620][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:26:32,945][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:26:33,271][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:26:33,597][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:26:33,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:26:34,251][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:26:34,578][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:26:34,902][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:26:35,227][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:26:35,553][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:26:35,878][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:26:36,204][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:26:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:26:36,860][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:26:37,189][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:26:37,514][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:26:37,840][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:26:38,166][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:26:38,492][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:26:38,818][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:26:39,144][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:26:39,471][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:26:39,798][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:26:40,125][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:26:40,456][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:26:40,783][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:26:41,111][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:26:41,447][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:26:41,773][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:26:42,099][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:26:42,425][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:26:43,176][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:26:43,914][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:26:43,916][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:26:43,917][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:26:44,829][__main__][INFO] - Iteration 379 took 23s (38.95% Gen, 57.12% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 59m 46s. Estimated total time: 19h 22m 42s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 45s, 500 more iterations: 3h 13m 47s.
[2025-11-13 10:26:44,831][__main__][INFO] - Starting iteration 379.
[2025-11-13 10:26:44,834][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:26:44,835][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:54,187][__main__][INFO] - Number of regex retries in iteration 379: 0
[2025-11-13 10:26:54,188][__main__][INFO] - agents played in iteration 379 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:26:54,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:54,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:54,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:54,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:54,785][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:54,785][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:55,570][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:55,870][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:56,196][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:26:56,522][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:26:56,848][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:26:57,174][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:26:57,501][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:26:57,827][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:26:58,152][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:26:58,479][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:26:58,805][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:26:59,131][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:26:59,457][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:26:59,783][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:00,109][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:00,437][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:00,766][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:01,095][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:01,421][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:01,748][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:02,075][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:02,402][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:02,727][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:03,054][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:03,381][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:03,706][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:04,032][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:04,360][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:04,687][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:05,018][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:05,344][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:05,674][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:06,001][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:06,744][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:07,452][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:07,454][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:07,455][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:08,378][__main__][INFO] - Iteration 380 took 23s (39.72% Gen, 56.35% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 13m 55s. Estimated total time: 19h 37m 14s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 12s.
[2025-11-13 10:27:08,380][__main__][INFO] - Starting iteration 380.
[2025-11-13 10:27:08,383][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:27:08,384][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:27:17,438][__main__][INFO] - Number of regex retries in iteration 380: 0
[2025-11-13 10:27:17,439][__main__][INFO] - agents played in iteration 380 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:27:17,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:17,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:18,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:18,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:18,049][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:27:18,049][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:27:18,808][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:27:19,105][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:27:19,430][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:27:19,756][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:27:20,085][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:27:20,410][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:27:20,737][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:27:21,064][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:27:21,389][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:27:21,715][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:27:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:27:22,366][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:27:22,691][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:27:23,016][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:23,342][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:23,666][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:23,995][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:24,321][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:24,646][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:24,972][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:25,300][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:25,625][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:25,950][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:26,275][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:26,607][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:26,934][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:27,267][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:27,588][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:27,920][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:28,247][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:28,583][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:28,903][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:29,229][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:29,974][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:30,662][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:30,664][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:30,666][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:32,468][__main__][INFO] - Iteration 381 took 24s (37.59% Gen, 54.92% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 34s. Estimated total time: 20h 4m 18s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 8s, 500 more iterations: 3h 20m 43s.
[2025-11-13 10:27:32,471][__main__][INFO] - Starting iteration 381.
[2025-11-13 10:27:32,474][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:27:32,475][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:27:42,435][__main__][INFO] - Number of regex retries in iteration 381: 0
[2025-11-13 10:27:42,436][__main__][INFO] - agents played in iteration 381 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:27:42,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:42,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:43,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:43,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:43,041][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:27:43,041][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:27:43,810][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:27:44,104][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:27:44,430][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:27:44,765][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:27:45,091][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:27:45,420][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:27:45,748][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:27:46,076][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:27:46,401][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:27:46,728][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:27:47,059][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:27:47,381][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:27:47,708][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:27:48,034][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:48,367][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:48,686][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:49,011][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:49,337][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:49,669][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:49,995][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:50,324][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:50,648][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:50,985][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:51,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:51,636][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:51,967][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:52,294][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:52,622][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:52,947][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:53,272][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:53,599][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:53,925][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:54,252][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:54,988][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:55,710][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:55,711][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:55,713][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:56,651][__main__][INFO] - Iteration 382 took 24s (41.20% Gen, 54.91% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 44m 46s. Estimated total time: 20h 8m 54s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 17s, 500 more iterations: 3h 21m 29s.
[2025-11-13 10:27:56,654][__main__][INFO] - Starting iteration 382.
[2025-11-13 10:27:56,657][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:27:56,657][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:05,782][__main__][INFO] - Number of regex retries in iteration 382: 0
[2025-11-13 10:28:05,783][__main__][INFO] - agents played in iteration 382 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:28:06,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:06,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:06,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:06,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:06,371][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:06,372][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:07,126][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:07,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:07,750][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:08,083][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:08,402][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:08,728][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:09,055][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:09,387][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:09,705][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:10,030][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:10,356][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:10,687][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:11,007][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:28:11,333][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:28:11,659][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:28:11,990][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:28:12,313][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:28:12,638][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:28:12,965][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:28:13,296][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:28:13,617][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:28:13,943][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:28:14,267][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:28:14,599][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:28:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:28:15,246][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:28:15,572][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:28:15,897][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:28:16,224][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:28:16,550][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:28:16,876][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:28:17,203][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:28:17,531][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:28:18,273][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:28:18,984][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:28:18,986][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:28:18,988][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:28:20,016][__main__][INFO] - Iteration 383 took 23s (39.06% Gen, 56.53% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 3m 31s. Estimated total time: 19h 28m 2s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 40s.
[2025-11-13 10:28:20,019][__main__][INFO] - Starting iteration 383.
[2025-11-13 10:28:20,022][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:28:20,022][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:29,632][__main__][INFO] - Number of regex retries in iteration 383: 0
[2025-11-13 10:28:29,633][__main__][INFO] - agents played in iteration 383 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:28:30,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:30,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:30,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:30,227][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:30,228][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:30,228][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:31,013][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:31,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:31,638][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:32,627][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:32,952][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:33,281][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:33,606][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:33,931][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:34,256][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:34,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:34,912][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:28:35,238][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:28:35,564][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:28:35,889][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:28:36,214][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:28:36,540][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:28:36,867][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:28:37,194][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:28:37,524][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:28:37,849][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:28:38,178][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:28:38,506][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:28:38,834][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:28:39,160][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:28:39,486][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:28:39,813][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:28:40,139][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:28:40,465][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:28:40,791][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:28:41,117][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:28:41,443][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:28:42,197][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:28:42,904][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:28:42,905][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:28:42,907][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:28:43,925][__main__][INFO] - Iteration 384 took 23s (40.20% Gen, 55.53% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 30m 19s. Estimated total time: 19h 55m 14s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 12s.
[2025-11-13 10:28:43,928][__main__][INFO] - Starting iteration 384.
[2025-11-13 10:28:43,930][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:28:43,931][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:53,045][__main__][INFO] - Number of regex retries in iteration 384: 0
[2025-11-13 10:28:53,046][__main__][INFO] - agents played in iteration 384 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:28:53,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:53,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:53,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:53,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:53,637][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:53,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:54,394][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:54,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:55,018][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:55,345][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:55,671][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:55,997][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:56,323][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:56,650][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:56,978][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:57,303][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:57,629][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:57,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:58,292][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:28:58,611][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:28:58,937][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:28:59,263][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:28:59,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:28:59,917][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:00,242][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:00,567][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:00,899][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:01,218][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:01,546][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:01,871][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:02,205][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:02,522][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:02,847][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:03,173][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:03,506][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:03,825][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:04,151][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:04,479][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:04,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:05,568][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:06,270][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:06,271][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:06,273][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:07,253][__main__][INFO] - Iteration 385 took 23s (39.08% Gen, 56.71% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 51s. Estimated total time: 19h 26m 10s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 21s.
[2025-11-13 10:29:07,255][__main__][INFO] - Starting iteration 385.
[2025-11-13 10:29:07,259][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:07,259][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:29:17,054][__main__][INFO] - Number of regex retries in iteration 385: 0
[2025-11-13 10:29:17,054][__main__][INFO] - agents played in iteration 385 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:29:17,537][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:17,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:17,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:17,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:17,637][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:29:17,638][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:29:18,396][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:29:18,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:29:19,019][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:29:19,346][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:29:19,672][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:29:19,997][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:29:20,323][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:29:20,651][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:29:20,986][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:29:21,316][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:29:21,642][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:29:21,968][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:29:22,304][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:29:22,633][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:29:22,959][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:29:23,284][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:29:23,610][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:29:23,941][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:24,270][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:24,601][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:24,927][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:25,578][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:26,230][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:26,555][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:26,880][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:27,205][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:27,531][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:27,858][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:28,186][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:28,513][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:28,837][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:29,574][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:30,304][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:30,306][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:30,308][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:31,317][__main__][INFO] - Iteration 386 took 24s (40.71% Gen, 55.09% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 37m 15s. Estimated total time: 20h 2m 57s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 5s, 500 more iterations: 3h 20m 29s.
[2025-11-13 10:29:31,319][__main__][INFO] - Starting iteration 386.
[2025-11-13 10:29:31,323][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:31,323][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:29:40,288][__main__][INFO] - Number of regex retries in iteration 386: 0
[2025-11-13 10:29:40,288][__main__][INFO] - agents played in iteration 386 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:29:40,791][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:40,827][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:40,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:40,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:40,895][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:29:40,895][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:29:41,645][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:29:41,942][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:29:42,270][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:29:42,596][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:29:42,923][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:29:43,249][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:29:43,577][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:29:43,905][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:29:44,231][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:29:44,557][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:29:44,883][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:29:45,218][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:29:45,548][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:29:45,875][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:29:46,207][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:29:46,529][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:29:46,856][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:29:47,185][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:47,521][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:47,843][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:48,170][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:48,496][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:48,828][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:49,148][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:49,473][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:49,800][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:50,132][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:50,451][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:50,777][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:51,104][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:51,430][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:51,756][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:52,081][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:52,839][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:53,556][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:53,558][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:53,560][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:54,566][__main__][INFO] - Iteration 387 took 23s (38.57% Gen, 57.10% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 56m 5s. Estimated total time: 19h 22m 11s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 41s.
[2025-11-13 10:29:54,568][__main__][INFO] - Starting iteration 387.
[2025-11-13 10:29:54,572][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:54,572][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:03,079][__main__][INFO] - Number of regex retries in iteration 387: 0
[2025-11-13 10:30:03,080][__main__][INFO] - agents played in iteration 387 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:30:03,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:03,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:04,000][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:04,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:04,035][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:04,035][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:04,806][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:05,102][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:05,429][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:05,754][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:06,079][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:06,404][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:06,730][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:07,056][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:07,382][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:07,708][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:08,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:08,359][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:08,684][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:09,010][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:09,338][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:09,664][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:09,995][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:10,321][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:10,645][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:30:10,971][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:30:11,300][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:30:11,626][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:30:11,952][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:30:12,278][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:30:12,606][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:30:12,932][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:30:13,258][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:30:13,584][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:30:13,915][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:30:14,243][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:30:14,569][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:30:14,895][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:30:15,223][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:30:15,964][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:30:16,718][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:30:16,720][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:30:16,722][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:30:17,710][__main__][INFO] - Iteration 388 took 23s (36.77% Gen, 58.95% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 50m 29s. Estimated total time: 19h 16m 58s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 49s.
[2025-11-13 10:30:17,712][__main__][INFO] - Starting iteration 388.
[2025-11-13 10:30:17,716][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:30:17,716][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:26,605][__main__][INFO] - Number of regex retries in iteration 388: 0
[2025-11-13 10:30:26,606][__main__][INFO] - agents played in iteration 388 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:30:27,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:27,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:27,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:27,197][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:27,198][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:27,198][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:27,940][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:28,241][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:28,565][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:28,890][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:29,216][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:29,541][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:29,866][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:30,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:30,845][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:31,171][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:31,496][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:31,823][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:32,150][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:32,476][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:32,804][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:33,128][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:33,455][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:33,781][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:30:34,113][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:30:34,439][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:30:34,766][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:30:35,099][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:30:35,425][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:30:35,752][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:30:36,081][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:30:36,406][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:30:36,734][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:30:37,059][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:30:37,384][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:30:37,714][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:30:38,035][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:30:38,360][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:30:39,131][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:30:39,875][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:30:39,877][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:30:39,879][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:30:40,912][__main__][INFO] - Iteration 389 took 23s (38.32% Gen, 57.22% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 53m 0s. Estimated total time: 19h 19m 52s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 39s, 500 more iterations: 3h 13m 18s.
[2025-11-13 10:30:40,915][__main__][INFO] - Starting iteration 389.
[2025-11-13 10:30:40,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:30:40,919][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:50,392][__main__][INFO] - Number of regex retries in iteration 389: 0
[2025-11-13 10:30:50,393][__main__][INFO] - agents played in iteration 389 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:30:50,886][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:50,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:50,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:50,988][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:50,989][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:50,989][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:51,737][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:52,034][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:52,361][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:52,688][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:53,018][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:53,343][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:53,667][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:53,993][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:54,320][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:54,644][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:54,971][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:55,300][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:55,626][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:55,952][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:56,277][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:56,602][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:56,928][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:57,253][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:57,579][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:30:57,905][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:30:58,230][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:30:58,556][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:30:58,884][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:30:59,211][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:30:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:30:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:00,193][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:00,519][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:00,845][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:01,172][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:01,502][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:01,828][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:02,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:02,954][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:03,690][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:03,692][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:03,693][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:04,743][__main__][INFO] - Iteration 390 took 23s (39.76% Gen, 55.82% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 24m 1s. Estimated total time: 19h 51m 17s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 42s, 500 more iterations: 3h 18m 32s.
[2025-11-13 10:31:04,745][__main__][INFO] - Starting iteration 390.
[2025-11-13 10:31:04,748][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:31:04,749][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:31:14,174][__main__][INFO] - Number of regex retries in iteration 390: 0
[2025-11-13 10:31:14,174][__main__][INFO] - agents played in iteration 390 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:31:14,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:14,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:14,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:14,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:14,768][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:31:14,769][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:31:15,532][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:31:15,835][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:31:16,161][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:31:16,487][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:31:16,815][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:31:17,140][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:31:17,466][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:31:17,795][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:31:18,120][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:31:18,447][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:31:18,772][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:31:19,100][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:31:19,424][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:31:19,750][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:31:20,076][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:31:20,401][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:31:20,728][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:31:21,052][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:31:21,378][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:31:21,707][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:31:22,033][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:31:22,359][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:31:22,685][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:31:23,010][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:31:23,335][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:31:23,663][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:23,989][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:24,314][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:24,641][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:24,966][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:25,292][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:25,617][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:25,943][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:26,707][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:27,410][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:27,412][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:27,413][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:29,697][__main__][INFO] - Iteration 391 took 24s (37.77% Gen, 53.06% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 19m 48s. Estimated total time: 20h 47m 29s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 34s, 500 more iterations: 3h 27m 54s.
[2025-11-13 10:31:29,699][__main__][INFO] - Starting iteration 391.
[2025-11-13 10:31:29,703][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:31:29,703][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:31:39,588][__main__][INFO] - Number of regex retries in iteration 391: 0
[2025-11-13 10:31:39,589][__main__][INFO] - agents played in iteration 391 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:31:40,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:40,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:40,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:40,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:40,188][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:31:40,188][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:31:40,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:31:41,258][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:31:41,584][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:31:41,909][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:31:42,239][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:31:42,564][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:31:42,890][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:31:43,217][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:31:43,548][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:31:43,873][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:31:44,200][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:31:44,530][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:31:44,856][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:31:45,182][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:31:45,513][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:31:45,845][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:31:46,164][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:31:46,493][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:31:46,819][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:31:47,145][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:31:47,471][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:31:47,796][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:31:48,122][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:31:48,447][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:31:48,772][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:31:49,097][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:49,422][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:49,748][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:50,074][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:50,399][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:50,724][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:51,051][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:51,379][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:52,133][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:52,884][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:52,885][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:52,887][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:53,799][__main__][INFO] - Iteration 392 took 24s (41.02% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 36m 47s. Estimated total time: 20h 4m 52s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 9s, 500 more iterations: 3h 20m 48s.
[2025-11-13 10:31:53,801][__main__][INFO] - Starting iteration 392.
[2025-11-13 10:31:53,804][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:31:53,804][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:02,926][__main__][INFO] - Number of regex retries in iteration 392: 0
[2025-11-13 10:32:02,927][__main__][INFO] - agents played in iteration 392 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:32:03,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:03,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:03,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:03,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:03,532][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:03,532][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:04,277][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:04,574][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:04,900][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:05,225][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:05,552][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:05,880][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:06,204][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:06,530][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:06,857][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:07,184][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:07,515][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:07,842][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:08,170][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:08,499][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:08,836][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:09,164][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:09,489][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:09,815][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:10,144][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:10,468][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:10,794][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:11,121][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:11,447][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:11,773][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:12,100][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:32:12,426][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:32:12,753][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:32:13,083][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:32:13,410][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:32:13,744][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:32:14,062][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:32:14,388][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:32:14,713][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:32:15,504][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:32:16,261][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:32:16,263][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:32:16,265][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:32:17,235][__main__][INFO] - Iteration 393 took 23s (38.93% Gen, 56.92% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 3m 7s. Estimated total time: 19h 31m 36s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 3s, 500 more iterations: 3h 15m 16s.
[2025-11-13 10:32:17,237][__main__][INFO] - Starting iteration 393.
[2025-11-13 10:32:17,241][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:32:17,242][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:26,649][__main__][INFO] - Number of regex retries in iteration 393: 0
[2025-11-13 10:32:26,649][__main__][INFO] - agents played in iteration 393 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:32:27,144][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:27,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:27,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:27,250][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:27,251][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:27,251][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:27,995][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:28,292][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:28,619][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:28,944][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:29,270][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:29,596][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:29,923][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:30,250][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:30,576][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:30,905][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:31,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:31,557][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:31,884][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:32,219][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:32,551][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:32,878][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:33,206][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:33,531][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:33,858][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:34,183][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:34,509][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:34,846][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:35,174][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:35,503][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:35,828][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:32:36,155][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:32:36,481][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:32:36,806][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:32:37,132][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:32:37,460][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:32:37,787][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:32:38,118][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:32:38,445][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:32:39,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:32:39,983][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:32:39,984][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:32:39,986][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:32:40,952][__main__][INFO] - Iteration 394 took 23s (39.67% Gen, 56.24% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 16m 44s. Estimated total time: 19h 45m 36s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 36s.
[2025-11-13 10:32:40,954][__main__][INFO] - Starting iteration 394.
[2025-11-13 10:32:40,957][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:32:40,957][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:50,803][__main__][INFO] - Number of regex retries in iteration 394: 0
[2025-11-13 10:32:50,803][__main__][INFO] - agents played in iteration 394 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:32:51,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:51,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:51,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:51,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:51,386][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:51,387][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:52,137][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:52,433][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:52,759][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:53,086][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:53,415][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:53,741][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:54,067][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:54,398][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:54,727][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:55,058][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:55,386][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:55,711][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:56,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:56,363][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:56,690][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:57,019][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:57,346][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:57,672][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:57,998][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:58,324][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:58,656][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:58,987][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:59,312][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:59,640][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:59,969][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:33:00,295][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:33:00,622][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:33:00,958][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:33:01,280][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:33:01,610][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:01,941][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:02,275][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:02,600][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:03,381][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:04,096][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:04,097][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:04,099][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:05,097][__main__][INFO] - Iteration 395 took 24s (40.79% Gen, 55.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 37m 48s. Estimated total time: 20h 7m 4s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 14s, 500 more iterations: 3h 21m 10s.
[2025-11-13 10:33:05,100][__main__][INFO] - Starting iteration 395.
[2025-11-13 10:33:05,102][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:05,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:33:13,872][__main__][INFO] - Number of regex retries in iteration 395: 0
[2025-11-13 10:33:13,873][__main__][INFO] - agents played in iteration 395 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:33:14,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:14,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:14,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:14,465][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:14,465][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:33:14,466][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:33:15,208][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:33:15,505][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:33:15,830][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:33:16,156][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:33:16,487][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:33:16,812][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:33:17,138][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:33:17,465][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:33:17,789][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:33:18,117][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:33:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:33:18,770][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:33:19,096][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:33:19,421][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:33:19,747][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:33:20,072][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:33:20,399][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:33:20,727][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:33:21,054][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:33:21,381][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:33:21,711][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:33:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:33:22,367][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:33:22,693][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:33:23,018][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:33:23,347][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:33:23,674][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:33:23,999][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:33:24,326][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:33:24,654][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:24,982][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:25,309][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:25,639][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:26,406][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:27,151][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:27,153][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:27,156][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:28,090][__main__][INFO] - Iteration 396 took 22s (38.15% Gen, 57.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 39m 46s. Estimated total time: 19h 9m 25s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 18s, 500 more iterations: 3h 11m 34s.
[2025-11-13 10:33:28,092][__main__][INFO] - Starting iteration 396.
[2025-11-13 10:33:28,117][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:28,118][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:33:36,841][__main__][INFO] - Number of regex retries in iteration 396: 0
[2025-11-13 10:33:36,841][__main__][INFO] - agents played in iteration 396 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:33:37,331][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,432][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:33:37,433][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:33:38,183][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:33:38,480][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:33:38,806][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:33:39,136][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:33:39,461][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:33:39,788][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:33:40,115][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:33:40,441][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:33:40,767][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:33:41,093][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:33:41,419][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:33:41,752][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:33:42,082][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:33:42,409][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:33:42,734][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:33:43,061][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:33:43,388][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:33:43,714][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:33:44,041][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:33:44,367][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:33:44,693][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:33:45,020][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:33:45,346][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:33:45,673][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:33:46,000][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:33:46,325][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:33:46,651][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:33:46,977][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:33:47,308][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:33:47,635][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:47,962][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:48,289][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:48,617][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:49,364][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:50,115][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:50,117][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:50,118][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:51,135][__main__][INFO] - Iteration 397 took 23s (37.86% Gen, 57.62% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 42m 1s. Estimated total time: 19h 12m 4s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 24s, 500 more iterations: 3h 12m 0s.
[2025-11-13 10:33:51,137][__main__][INFO] - Starting iteration 397.
[2025-11-13 10:33:51,141][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:51,141][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:00,571][__main__][INFO] - Number of regex retries in iteration 397: 0
[2025-11-13 10:34:00,572][__main__][INFO] - agents played in iteration 397 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:34:01,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:01,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:01,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:01,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:01,168][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:01,169][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:01,916][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:02,213][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:02,540][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:02,870][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:03,196][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:03,521][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:03,847][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:04,174][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:04,499][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:04,830][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:05,157][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:05,484][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:05,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:06,138][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:06,463][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:06,792][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:07,120][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:07,446][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:07,777][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:08,106][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:08,432][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:08,759][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:09,089][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:09,416][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:09,741][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:10,065][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:10,390][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:10,717][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:11,042][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:34:11,368][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:34:11,694][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:34:12,020][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:34:12,349][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:34:13,100][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:34:14,013][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:34:14,014][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:34:14,016][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:34:14,987][__main__][INFO] - Iteration 398 took 23s (39.54% Gen, 56.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 21m 57s. Estimated total time: 19h 52m 23s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 44s, 500 more iterations: 3h 18m 43s.
[2025-11-13 10:34:14,989][__main__][INFO] - Starting iteration 398.
[2025-11-13 10:34:14,993][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:34:14,993][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:24,384][__main__][INFO] - Number of regex retries in iteration 398: 0
[2025-11-13 10:34:24,385][__main__][INFO] - agents played in iteration 398 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:34:24,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,981][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,981][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:24,983][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:25,772][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:26,067][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:26,394][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:26,720][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:27,048][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:27,374][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:27,700][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:28,026][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:28,351][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:28,677][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:29,002][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:29,328][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:29,653][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:29,979][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:30,306][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:30,632][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:30,958][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:31,285][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:31,610][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:31,936][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:32,261][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:32,587][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:32,912][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:33,237][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:33,564][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:33,889][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:34,214][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:34,540][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:34,866][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:34:35,190][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:34:35,521][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:34:35,846][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:34:36,174][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:34:36,961][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:34:37,704][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:34:37,706][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:34:37,708][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:34:38,779][__main__][INFO] - Iteration 399 took 23s (39.48% Gen, 56.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 18m 30s. Estimated total time: 19h 49m 20s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 38s, 500 more iterations: 3h 18m 13s.
[2025-11-13 10:34:38,781][__main__][INFO] - Starting iteration 399.
[2025-11-13 10:34:38,784][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:34:38,785][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:48,134][__main__][INFO] - Number of regex retries in iteration 399: 0
[2025-11-13 10:34:48,135][__main__][INFO] - agents played in iteration 399 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:34:48,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:48,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:48,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:48,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:48,743][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:48,743][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:49,548][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:49,845][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:50,175][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:50,501][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:50,828][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:51,154][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:51,481][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:51,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:52,131][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:52,456][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:52,783][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:53,112][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:53,438][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:53,763][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:54,090][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:54,415][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:54,741][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:55,068][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:55,395][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:55,721][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:56,046][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:56,371][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:56,698][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:57,024][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:57,350][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:57,676][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:58,003][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:58,329][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:58,657][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:34:58,986][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:34:59,313][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:34:59,640][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:34:59,968][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:00,730][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:01,454][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:01,455][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:01,458][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:02,548][__main__][INFO] - Iteration 400 took 23s (39.34% Gen, 56.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 16m 59s. Estimated total time: 19h 48m 12s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 36s, 500 more iterations: 3h 18m 2s.
[2025-11-13 10:35:02,550][__main__][INFO] - Starting iteration 400.
[2025-11-13 10:35:02,553][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:35:02,554][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:35:11,966][__main__][INFO] - Number of regex retries in iteration 400: 0
[2025-11-13 10:35:11,967][__main__][INFO] - agents played in iteration 400 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:35:12,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,580][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:35:12,580][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:35:13,370][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:35:13,668][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:35:13,994][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:35:14,320][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:35:14,645][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:35:14,973][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:35:15,301][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:35:15,627][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:35:15,954][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:35:16,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:35:16,605][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:35:16,931][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:35:17,257][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:35:17,588][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:35:17,909][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:35:18,234][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:35:18,560][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:35:18,897][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:35:19,215][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:35:19,540][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:35:19,866][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:35:20,193][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:35:20,520][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:35:20,844][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:35:21,170][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:35:21,495][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:35:21,820][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:35:22,146][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:35:22,471][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:35:22,798][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:35:23,123][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:35:23,448][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:35:23,773][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:24,536][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:25,270][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:25,272][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:25,273][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:27,380][__main__][INFO] - Iteration 401 took 24s (37.91% Gen, 53.60% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 9m 46s. Estimated total time: 20h 41m 24s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 22s, 500 more iterations: 3h 26m 54s.
[2025-11-13 10:35:27,385][__main__][INFO] - Starting iteration 401.
[2025-11-13 10:35:27,388][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:35:27,389][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:35:36,796][__main__][INFO] - Number of regex retries in iteration 401: 0
[2025-11-13 10:35:36,797][__main__][INFO] - agents played in iteration 401 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:35:37,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:37,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:37,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:37,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:37,434][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:35:37,434][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:35:38,202][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:35:38,498][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:35:38,825][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:35:39,149][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:35:39,475][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:35:39,802][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:35:40,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:35:40,454][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:35:40,779][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:35:41,107][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:35:41,436][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:35:41,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:35:42,088][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:35:42,414][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:35:42,739][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:35:43,064][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:35:43,391][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:35:43,718][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:35:44,044][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:35:44,369][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:35:44,696][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:35:45,026][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:35:45,357][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:35:45,686][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:35:46,011][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:35:46,336][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:35:46,668][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:35:46,990][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:35:47,317][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:35:47,642][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:35:47,971][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:35:48,297][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:35:48,623][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:49,402][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:50,139][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:50,145][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:50,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:51,225][__main__][INFO] - Iteration 402 took 23s (39.47% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 19m 52s. Estimated total time: 19h 51m 54s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 39s.
[2025-11-13 10:35:51,227][__main__][INFO] - Starting iteration 402.
[2025-11-13 10:35:51,230][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:35:51,230][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:36:00,443][__main__][INFO] - Number of regex retries in iteration 402: 0
[2025-11-13 10:36:00,444][__main__][INFO] - agents played in iteration 402 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:36:00,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:00,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:01,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:01,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:01,065][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:36:01,066][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:01,840][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:02,137][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:02,463][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:02,799][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:03,124][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:03,449][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:03,774][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:04,100][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:04,425][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:04,750][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:05,076][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:05,405][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:05,730][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:06,055][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:06,381][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:06,709][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:07,035][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:07,362][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:07,689][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:08,016][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:08,347][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:08,677][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:09,002][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:09,327][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:09,653][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:09,977][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:10,301][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:10,627][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:10,954][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:11,279][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:11,605][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:11,931][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:12,255][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:36:12,995][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:36:13,732][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:36:13,736][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:36:13,738][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:36:14,664][__main__][INFO] - Iteration 403 took 23s (39.31% Gen, 56.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 59m 20s. Estimated total time: 19h 31m 46s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 3s, 500 more iterations: 3h 15m 17s.
[2025-11-13 10:36:14,667][__main__][INFO] - Starting iteration 403.
[2025-11-13 10:36:14,670][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:36:14,671][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:36:23,907][__main__][INFO] - Number of regex retries in iteration 403: 0
[2025-11-13 10:36:23,907][__main__][INFO] - agents played in iteration 403 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:36:24,410][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:24,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:24,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:24,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:24,512][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:36:24,513][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:25,294][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:25,590][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:25,917][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:26,243][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:26,568][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:26,892][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:27,218][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:27,544][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:27,870][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:28,197][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:28,523][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:28,848][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:29,174][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:29,499][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:29,824][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:30,149][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:30,479][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:30,807][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:31,133][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:31,464][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:31,791][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:32,116][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:32,440][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:32,769][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:33,095][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:33,421][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:33,748][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:34,075][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:34,725][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:35,050][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:35,375][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:35,703][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:36:36,443][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:36:37,196][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:36:37,198][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:36:37,200][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:36:38,265][__main__][INFO] - Iteration 404 took 23s (39.14% Gen, 56.33% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 6m 57s. Estimated total time: 19h 39m 47s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 37s.
[2025-11-13 10:36:38,267][__main__][INFO] - Starting iteration 404.
[2025-11-13 10:36:38,270][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:36:38,270][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:36:47,759][__main__][INFO] - Number of regex retries in iteration 404: 0
[2025-11-13 10:36:47,760][__main__][INFO] - agents played in iteration 404 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:36:48,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:48,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:48,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:48,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:48,353][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:36:48,353][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:49,116][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:49,411][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:49,737][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:50,063][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:50,387][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:50,712][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:51,041][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:51,365][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:51,691][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:52,017][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:52,342][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:52,667][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:52,992][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:53,318][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:53,648][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:53,977][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:54,308][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:54,639][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:54,967][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:55,293][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:55,620][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:55,950][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:56,280][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:56,610][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:56,939][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:57,265][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:57,601][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:57,926][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:58,257][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:58,588][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:58,919][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:59,579][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:37:00,313][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:37:01,030][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:37:01,032][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:37:01,033][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:01,942][__main__][INFO] - Iteration 405 took 23s (40.09% Gen, 56.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 27s. Estimated total time: 19h 43m 40s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 27s, 500 more iterations: 3h 17m 16s.
[2025-11-13 10:37:01,944][__main__][INFO] - Starting iteration 405.
[2025-11-13 10:37:01,947][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:01,948][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:10,568][__main__][INFO] - Number of regex retries in iteration 405: 0
[2025-11-13 10:37:10,568][__main__][INFO] - agents played in iteration 405 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:37:11,056][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:11,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:11,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:11,157][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:11,157][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:11,158][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:11,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:12,235][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:12,560][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:12,886][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:37:13,218][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:37:13,544][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:37:13,869][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:37:14,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:37:14,518][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:37:14,845][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:37:15,171][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:37:15,497][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:37:15,826][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:37:16,160][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:37:16,485][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:37:16,814][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:37:17,135][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:37:17,460][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:37:17,786][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:37:18,116][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:37:18,437][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:37:18,763][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:37:19,089][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:37:19,418][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:37:19,739][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:37:20,067][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:37:20,395][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:37:20,723][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:37:21,051][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:37:21,380][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:37:21,706][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:37:22,035][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:37:22,368][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:37:23,100][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:37:23,816][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:37:23,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:37:23,822][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:24,819][__main__][INFO] - Iteration 406 took 22s (37.69% Gen, 57.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 30m 4s. Estimated total time: 19h 3m 40s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 7s, 500 more iterations: 3h 10m 36s.
[2025-11-13 10:37:24,821][__main__][INFO] - Starting iteration 406.
[2025-11-13 10:37:24,824][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:24,825][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:33,809][__main__][INFO] - Number of regex retries in iteration 406: 0
[2025-11-13 10:37:33,809][__main__][INFO] - agents played in iteration 406 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:37:34,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:34,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:34,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:34,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:34,768][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:34,769][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:35,548][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:35,913][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:36,238][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:36,562][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:37:36,887][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:37:37,212][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:37:37,538][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:37:37,864][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:37:38,195][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:37:38,521][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:37:38,849][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:37:39,176][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:37:39,501][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:37:39,826][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:37:40,151][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:37:40,477][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:37:40,804][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:37:41,129][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:37:41,454][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:37:41,781][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:37:42,108][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:37:42,436][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:37:42,762][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:37:43,089][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:37:43,416][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:37:43,742][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:37:44,067][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:37:44,393][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:37:44,720][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:37:45,047][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:37:45,374][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:37:45,700][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:37:46,026][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:37:46,748][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:37:47,479][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:37:47,480][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:37:47,482][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:48,397][__main__][INFO] - Iteration 407 took 23s (38.11% Gen, 58.00% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 4m 40s. Estimated total time: 19h 38m 40s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 26s.
[2025-11-13 10:37:48,399][__main__][INFO] - Starting iteration 407.
[2025-11-13 10:37:48,402][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:48,403][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:57,400][__main__][INFO] - Number of regex retries in iteration 407: 0
[2025-11-13 10:37:57,400][__main__][INFO] - agents played in iteration 407 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:37:57,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:57,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:57,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:58,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:58,005][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:58,006][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:58,778][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:59,075][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:59,400][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:59,725][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:38:00,051][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:38:00,376][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:38:00,706][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:01,032][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:01,358][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:01,685][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:02,017][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:02,343][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:02,669][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:02,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:03,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:03,650][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:03,981][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:04,308][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:04,635][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:04,960][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:05,288][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:05,615][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:05,946][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:06,275][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:06,603][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:06,930][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:07,259][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:07,586][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:07,914][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:08,242][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:08,574][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:08,900][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:09,226][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:09,980][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:10,717][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:10,719][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:10,721][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:11,707][__main__][INFO] - Iteration 408 took 23s (38.61% Gen, 57.16% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 50m 53s. Estimated total time: 19h 25m 16s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 50s, 500 more iterations: 3h 14m 12s.
[2025-11-13 10:38:11,709][__main__][INFO] - Starting iteration 408.
[2025-11-13 10:38:11,712][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:38:11,712][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:38:21,098][__main__][INFO] - Number of regex retries in iteration 408: 0
[2025-11-13 10:38:21,099][__main__][INFO] - agents played in iteration 408 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:38:21,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:21,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:21,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:21,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:21,704][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:38:21,705][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:38:22,477][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:38:22,772][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:38:23,098][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:38:23,424][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:38:23,750][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:38:24,075][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:38:24,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:24,728][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:25,058][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:25,385][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:25,713][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:26,038][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:26,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:26,689][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:27,016][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:27,343][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:27,671][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:27,998][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:28,325][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:28,651][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:28,978][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:29,304][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:29,636][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:29,966][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:30,291][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:30,941][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:31,266][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:31,591][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:31,922][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:32,906][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:33,654][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:34,382][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:34,384][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:34,386][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:35,406][__main__][INFO] - Iteration 409 took 23s (39.61% Gen, 56.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 0s. Estimated total time: 19h 44m 47s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 29s, 500 more iterations: 3h 17m 27s.
[2025-11-13 10:38:35,409][__main__][INFO] - Starting iteration 409.
[2025-11-13 10:38:35,412][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:38:35,412][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:38:44,240][__main__][INFO] - Number of regex retries in iteration 409: 0
[2025-11-13 10:38:44,241][__main__][INFO] - agents played in iteration 409 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:38:44,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:44,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:44,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:44,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:44,837][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:38:44,837][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:38:45,635][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:38:45,932][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:38:46,259][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:38:46,585][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:38:46,912][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:38:47,239][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:38:47,565][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:47,890][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:48,216][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:48,542][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:48,870][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:49,196][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:49,522][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:49,847][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:50,174][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:50,501][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:50,830][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:51,156][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:51,480][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:51,807][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:52,146][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:52,472][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:52,799][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:53,461][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:54,124][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:54,455][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:54,778][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:55,102][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:55,429][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:55,757][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:56,076][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:56,778][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:57,540][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:57,541][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:57,543][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:58,793][__main__][INFO] - Iteration 410 took 23s (37.76% Gen, 56.89% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 53m 57s. Estimated total time: 19h 29m 7s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 51s.
[2025-11-13 10:38:58,795][__main__][INFO] - Starting iteration 410.
[2025-11-13 10:38:58,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:38:58,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:08,366][__main__][INFO] - Number of regex retries in iteration 410: 0
[2025-11-13 10:39:08,366][__main__][INFO] - agents played in iteration 410 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:39:08,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:08,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:08,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:08,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:08,966][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:08,967][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:09,783][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:10,081][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:10,408][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:10,735][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:11,060][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:11,386][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:11,712][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:39:12,041][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:39:12,366][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:39:12,692][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:39:13,019][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:39:13,348][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:39:13,672][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:39:13,999][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:39:14,328][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:39:14,661][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:39:14,987][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:39:15,313][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:39:15,640][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:39:15,976][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:39:16,295][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:39:16,622][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:39:16,948][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:39:17,280][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:39:17,601][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:39:17,929][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:39:18,255][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:39:18,585][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:39:18,908][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:39:19,234][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:39:19,559][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:39:19,887][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:39:20,209][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:39:20,937][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:39:21,692][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:39:21,693][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:39:21,695][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:39:23,667][__main__][INFO] - Iteration 411 took 24s (38.47% Gen, 53.59% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 7m 51s. Estimated total time: 20h 43m 26s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 26s, 500 more iterations: 3h 27m 14s.
[2025-11-13 10:39:23,669][__main__][INFO] - Starting iteration 411.
[2025-11-13 10:39:23,671][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:39:23,672][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:33,378][__main__][INFO] - Number of regex retries in iteration 411: 0
[2025-11-13 10:39:33,379][__main__][INFO] - agents played in iteration 411 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:39:33,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:33,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:33,929][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:33,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:33,963][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:33,964][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:34,729][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:35,027][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:35,352][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:35,685][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:36,005][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:36,331][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:36,657][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:39:36,982][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:39:37,308][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:39:37,634][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:39:37,958][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:39:38,289][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:39:38,611][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:39:38,935][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:39:39,260][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:39:39,589][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:39:39,910][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:39:40,238][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:39:40,565][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:39:40,893][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:39:41,218][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:39:41,551][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:39:41,879][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:39:42,210][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:39:42,538][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:39:42,868][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:39:43,192][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:39:43,518][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:39:43,845][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:39:44,170][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:39:44,497][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:39:44,825][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:39:45,153][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:39:45,886][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:39:46,695][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:39:46,697][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:39:46,698][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:39:47,714][__main__][INFO] - Iteration 412 took 24s (40.37% Gen, 55.40% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 26m 11s. Estimated total time: 20h 2m 10s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 4s, 500 more iterations: 3h 20m 21s.
[2025-11-13 10:39:47,716][__main__][INFO] - Starting iteration 412.
[2025-11-13 10:39:47,718][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:39:47,719][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:56,593][__main__][INFO] - Number of regex retries in iteration 412: 0
[2025-11-13 10:39:56,594][__main__][INFO] - agents played in iteration 412 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:39:57,105][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:57,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:57,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:57,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:57,207][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:57,208][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:57,963][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:58,262][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:58,589][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:58,915][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:59,241][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:59,569][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:59,895][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:40:00,221][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:40:00,547][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:40:00,877][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:01,205][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:01,533][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:01,858][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:02,186][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:02,512][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:02,838][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:03,165][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:03,492][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:03,818][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:04,147][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:04,479][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:04,804][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:05,136][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:05,468][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:05,796][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:06,127][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:06,458][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:06,783][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:07,108][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:07,433][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:07,758][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:08,084][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:08,409][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:09,152][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:09,937][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:09,939][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:09,940][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:11,004][__main__][INFO] - Iteration 413 took 23s (38.11% Gen, 57.31% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 47m 58s. Estimated total time: 19h 24m 21s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 3s.
[2025-11-13 10:40:11,006][__main__][INFO] - Starting iteration 413.
[2025-11-13 10:40:11,009][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:11,010][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:40:20,109][__main__][INFO] - Number of regex retries in iteration 413: 0
[2025-11-13 10:40:20,110][__main__][INFO] - agents played in iteration 413 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:40:20,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:20,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:20,681][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:20,715][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:20,716][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:40:20,716][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:40:21,467][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:40:21,777][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:40:22,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:40:22,431][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:40:22,758][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:40:23,087][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:40:23,413][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:40:23,738][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:40:24,066][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:40:24,393][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:24,718][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:25,043][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:25,369][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:25,695][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:26,023][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:26,347][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:26,675][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:27,001][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:27,327][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:27,652][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:27,977][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:28,304][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:28,630][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:28,955][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:29,280][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:29,608][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:30,258][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:30,585][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:30,911][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:31,236][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:31,561][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:31,887][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:32,612][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:33,345][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:33,346][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:33,348][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:34,364][__main__][INFO] - Iteration 414 took 23s (38.96% Gen, 56.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 2s. Estimated total time: 19h 27m 47s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 37s.
[2025-11-13 10:40:34,366][__main__][INFO] - Starting iteration 414.
[2025-11-13 10:40:34,369][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:34,369][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:40:43,653][__main__][INFO] - Number of regex retries in iteration 414: 0
[2025-11-13 10:40:43,654][__main__][INFO] - agents played in iteration 414 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:40:44,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:44,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:44,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:44,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:44,257][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:40:44,257][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:40:45,023][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:40:45,320][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:40:45,646][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:40:45,972][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:40:46,298][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:40:46,625][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:40:46,955][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:40:47,283][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:40:47,609][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:40:47,934][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:48,260][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:48,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:48,913][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:49,240][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:49,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:49,891][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:50,218][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:50,870][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:51,193][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:51,519][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:51,846][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:52,174][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:52,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:52,830][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:53,156][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:53,484][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:53,810][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:54,135][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:54,794][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:55,122][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:55,452][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:56,208][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:56,952][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:56,953][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:56,959][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:57,939][__main__][INFO] - Iteration 415 took 23s (39.39% Gen, 56.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 1m 22s. Estimated total time: 19h 38m 31s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 25s.
[2025-11-13 10:40:57,940][__main__][INFO] - Starting iteration 415.
[2025-11-13 10:40:57,943][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:57,944][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:07,750][__main__][INFO] - Number of regex retries in iteration 415: 0
[2025-11-13 10:41:07,750][__main__][INFO] - agents played in iteration 415 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:41:08,232][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:08,265][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:08,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:08,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:08,333][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:08,333][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:09,098][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:09,394][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:09,719][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:10,044][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:10,380][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:10,706][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:11,031][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:11,358][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:11,692][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:12,017][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:12,343][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:12,672][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:41:12,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:41:13,324][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:41:13,653][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:41:13,979][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:41:14,304][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:41:14,629][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:41:14,954][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:41:15,281][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:41:15,606][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:41:15,937][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:41:16,264][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:41:16,590][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:41:16,915][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:41:17,243][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:41:17,568][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:41:17,897][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:41:18,225][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:41:18,551][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:41:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:41:19,203][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:41:19,528][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:41:20,275][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:41:21,001][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:41:21,003][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:41:21,006][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:41:22,082][__main__][INFO] - Iteration 416 took 24s (40.62% Gen, 54.91% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 29m 25s. Estimated total time: 20h 6m 58s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 13s, 500 more iterations: 3h 21m 9s.
[2025-11-13 10:41:22,084][__main__][INFO] - Starting iteration 416.
[2025-11-13 10:41:22,088][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:41:22,088][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:31,699][__main__][INFO] - Number of regex retries in iteration 416: 0
[2025-11-13 10:41:31,700][__main__][INFO] - agents played in iteration 416 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:41:32,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:32,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:32,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:32,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:32,280][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:32,280][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:33,069][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:33,366][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:33,692][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:34,021][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:34,345][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:34,670][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:34,994][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:35,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:35,647][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:35,979][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:36,303][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:36,628][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:41:36,954][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:41:37,282][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:41:37,609][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:41:37,939][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:41:38,267][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:41:38,594][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:41:38,920][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:41:39,250][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:41:39,575][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:41:39,899][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:41:40,226][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:41:40,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:41:40,879][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:41:41,206][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:41:41,532][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:41:41,858][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:41:42,183][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:41:42,508][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:41:42,833][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:41:43,160][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:41:43,485][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:41:44,242][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:41:44,927][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:41:44,929][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:41:44,930][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:41:45,847][__main__][INFO] - Iteration 417 took 23s (40.45% Gen, 55.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 5s. Estimated total time: 19h 48m 2s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 36s, 500 more iterations: 3h 18m 0s.
[2025-11-13 10:41:45,849][__main__][INFO] - Starting iteration 417.
[2025-11-13 10:41:45,852][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:41:45,853][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:54,994][__main__][INFO] - Number of regex retries in iteration 417: 0
[2025-11-13 10:41:54,995][__main__][INFO] - agents played in iteration 417 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:41:55,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:55,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:55,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:55,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:55,581][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:55,581][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:56,362][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:56,659][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:56,985][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:57,317][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:57,637][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:57,963][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:58,292][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:58,626][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:58,954][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:59,282][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:59,938][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:42:00,270][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:42:00,599][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:42:00,924][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:42:01,255][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:42:01,589][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:42:01,916][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:02,240][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:02,568][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:02,898][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:03,230][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:03,556][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:03,883][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:04,210][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:04,537][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:04,862][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:05,190][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:05,517][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:05,847][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:06,174][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:06,501][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:06,832][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:07,595][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:08,322][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:08,323][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:08,325][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:09,244][__main__][INFO] - Iteration 418 took 23s (39.08% Gen, 56.99% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 18s. Estimated total time: 19h 29m 38s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 59s, 500 more iterations: 3h 14m 56s.
[2025-11-13 10:42:09,246][__main__][INFO] - Starting iteration 418.
[2025-11-13 10:42:09,249][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:42:09,250][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:42:18,639][__main__][INFO] - Number of regex retries in iteration 418: 0
[2025-11-13 10:42:18,640][__main__][INFO] - agents played in iteration 418 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:42:19,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:19,166][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:19,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:19,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:19,234][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:42:19,234][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:42:20,005][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:42:20,300][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:42:20,627][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:42:20,954][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:42:21,279][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:42:21,606][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:42:21,932][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:42:22,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:42:22,582][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:42:22,910][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:42:23,235][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:42:23,560][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:42:23,885][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:42:24,215][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:42:24,545][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:42:24,869][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:42:25,207][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:42:25,529][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:25,859][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:26,186][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:26,512][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:26,840][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:27,171][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:27,501][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:27,830][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:28,154][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:28,479][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:28,808][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:29,134][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:29,464][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:29,791][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:30,117][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:30,446][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:31,185][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:31,952][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:31,954][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:31,956][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:32,874][__main__][INFO] - Iteration 419 took 23s (39.74% Gen, 56.36% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 2m 35s. Estimated total time: 19h 41m 19s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 22s, 500 more iterations: 3h 16m 53s.
[2025-11-13 10:42:32,877][__main__][INFO] - Starting iteration 419.
[2025-11-13 10:42:32,879][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:42:32,880][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:42:42,655][__main__][INFO] - Number of regex retries in iteration 419: 0
[2025-11-13 10:42:42,656][__main__][INFO] - agents played in iteration 419 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:42:43,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:43,173][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:43,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:43,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:43,240][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:42:43,241][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:42:44,013][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:42:44,309][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:42:44,635][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:42:44,961][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:42:45,287][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:42:45,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:42:45,939][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:42:46,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:42:46,590][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:42:46,916][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:42:47,241][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:42:47,568][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:42:47,893][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:42:48,221][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:42:48,547][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:42:48,889][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:42:49,215][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:42:49,543][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:49,867][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:50,204][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:50,532][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:50,856][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:51,183][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:51,834][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:52,486][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:52,815][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:53,142][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:53,467][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:53,792][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:54,120][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:54,446][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:55,171][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:55,870][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:55,872][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:55,873][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:56,950][__main__][INFO] - Iteration 420 took 24s (40.61% Gen, 54.91% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 24m 26s. Estimated total time: 20h 3m 34s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 7s, 500 more iterations: 3h 20m 35s.
[2025-11-13 10:42:56,952][__main__][INFO] - Starting iteration 420.
[2025-11-13 10:42:56,955][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:42:56,955][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:06,235][__main__][INFO] - Number of regex retries in iteration 420: 0
[2025-11-13 10:43:06,235][__main__][INFO] - agents played in iteration 420 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:43:06,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:06,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:06,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:06,830][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:06,830][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:06,831][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:07,603][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:07,900][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:08,225][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:08,550][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:08,876][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:09,202][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:09,529][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:09,856][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:10,182][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:10,509][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:10,835][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:11,160][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:43:11,486][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:43:11,812][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:43:12,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:43:12,464][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:43:12,794][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:43:13,120][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:43:13,446][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:43:13,773][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:43:14,099][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:43:14,426][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:43:14,752][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:43:15,077][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:43:15,403][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:43:15,728][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:43:16,054][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:43:16,379][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:43:16,705][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:43:17,031][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:43:17,358][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:43:17,683][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:43:18,009][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:43:18,752][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:43:19,471][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:43:19,474][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:43:19,475][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:43:21,326][__main__][INFO] - Iteration 421 took 24s (38.08% Gen, 54.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 39m 3s. Estimated total time: 20h 18m 36s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 37s, 500 more iterations: 3h 23m 6s.
[2025-11-13 10:43:21,328][__main__][INFO] - Starting iteration 421.
[2025-11-13 10:43:21,331][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:43:21,331][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:31,461][__main__][INFO] - Number of regex retries in iteration 421: 0
[2025-11-13 10:43:31,462][__main__][INFO] - agents played in iteration 421 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:43:31,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:31,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:32,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:32,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:32,061][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:32,062][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:32,824][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:33,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:33,446][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:33,771][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:34,096][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:34,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:34,750][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:35,077][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:35,404][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:35,732][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:36,054][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:36,381][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:43:36,709][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:43:37,035][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:43:37,362][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:43:37,687][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:43:38,013][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:43:38,341][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:43:38,670][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:43:38,995][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:43:39,321][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:43:39,652][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:43:39,978][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:43:40,303][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:43:40,628][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:43:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:43:41,282][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:43:41,607][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:43:41,934][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:43:42,261][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:43:42,595][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:43:42,922][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:43:43,249][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:43:44,006][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:43:44,696][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:43:44,700][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:43:44,701][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:43:45,798][__main__][INFO] - Iteration 422 took 24s (41.40% Gen, 54.11% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 43m 27s. Estimated total time: 20h 23m 23s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 46s, 500 more iterations: 3h 23m 53s.
[2025-11-13 10:43:45,800][__main__][INFO] - Starting iteration 422.
[2025-11-13 10:43:45,803][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:43:45,804][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:54,902][__main__][INFO] - Number of regex retries in iteration 422: 0
[2025-11-13 10:43:54,902][__main__][INFO] - agents played in iteration 422 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:43:55,396][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:55,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:55,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:55,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:55,503][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:55,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:56,291][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:56,588][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:56,921][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:57,240][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:57,566][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:57,892][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:58,225][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:58,544][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:58,871][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:59,202][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:59,532][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:59,857][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:44:00,186][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:44:00,512][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:44:00,846][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:44:01,169][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:44:01,494][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:44:01,826][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:44:02,158][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:44:02,490][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:02,815][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:03,146][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:03,476][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:03,812][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:04,137][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:04,469][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:04,794][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:05,125][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:05,453][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:05,782][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:06,117][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:06,439][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:06,767][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:07,533][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:08,241][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:08,245][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:08,248][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:09,222][__main__][INFO] - Iteration 423 took 23s (38.85% Gen, 56.99% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 50m 40s. Estimated total time: 19h 31m 1s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 2s, 500 more iterations: 3h 15m 10s.
[2025-11-13 10:44:09,224][__main__][INFO] - Starting iteration 423.
[2025-11-13 10:44:09,227][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:09,228][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:44:18,008][__main__][INFO] - Number of regex retries in iteration 423: 0
[2025-11-13 10:44:18,010][__main__][INFO] - agents played in iteration 423 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:44:18,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:18,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:18,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:18,613][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:18,613][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:44:18,614][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:44:19,397][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:44:19,693][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:44:20,019][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:44:20,344][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:44:20,669][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:44:20,995][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:44:21,323][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:44:21,650][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:44:21,977][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:44:22,304][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:44:22,629][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:44:22,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:44:23,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:44:23,609][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:44:23,939][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:44:24,270][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:44:24,596][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:44:24,921][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:44:25,247][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:44:25,573][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:26,232][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:26,561][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:26,891][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:27,552][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:27,879][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:28,207][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:28,535][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:28,866][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:29,521][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:29,849][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:30,587][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:31,309][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:31,310][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:31,312][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:32,330][__main__][INFO] - Iteration 424 took 23s (38.01% Gen, 57.58% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 34m 26s. Estimated total time: 19h 15m 9s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 31s.
[2025-11-13 10:44:32,332][__main__][INFO] - Starting iteration 424.
[2025-11-13 10:44:32,335][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:32,336][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:44:42,011][__main__][INFO] - Number of regex retries in iteration 424: 0
[2025-11-13 10:44:42,011][__main__][INFO] - agents played in iteration 424 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:44:42,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:42,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:42,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:42,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:42,592][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:44:42,593][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:44:43,352][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:44:43,648][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:44:43,975][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:44:44,308][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:44:44,635][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:44:44,960][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:44:45,286][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:44:45,622][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:44:45,948][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:44:46,276][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:44:46,605][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:44:46,933][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:44:47,260][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:44:47,592][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:44:47,925][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:44:48,245][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:44:48,572][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:44:48,898][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:44:49,224][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:44:49,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:49,878][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:50,204][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:50,532][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:50,857][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:51,183][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:51,510][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:51,839][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:52,164][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:52,491][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:52,817][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:53,144][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:53,469][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:53,797][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:54,548][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:55,243][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:55,247][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:55,249][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:56,300][__main__][INFO] - Iteration 425 took 23s (40.37% Gen, 55.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 17m 11s. Estimated total time: 19h 58m 18s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 56s, 500 more iterations: 3h 19m 43s.
[2025-11-13 10:44:56,302][__main__][INFO] - Starting iteration 425.
[2025-11-13 10:44:56,305][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:56,306][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:05,066][__main__][INFO] - Number of regex retries in iteration 425: 0
[2025-11-13 10:45:05,067][__main__][INFO] - agents played in iteration 425 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:45:05,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:05,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:05,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:05,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:05,658][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:05,659][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:06,424][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:06,722][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:07,049][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:07,375][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:07,701][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:08,352][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:08,679][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:09,006][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:09,334][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:09,661][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:09,988][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:10,313][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:10,644][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:10,971][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:11,301][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:11,633][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:11,953][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:12,280][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:12,608][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:12,943][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:45:13,268][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:45:13,599][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:45:13,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:45:14,255][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:45:14,583][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:45:14,907][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:45:15,232][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:45:15,558][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:45:15,887][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:45:16,213][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:45:16,539][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:45:16,866][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:45:17,625][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:45:18,344][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:45:18,346][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:45:18,351][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:45:19,256][__main__][INFO] - Iteration 426 took 22s (38.17% Gen, 57.88% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 26m 2s. Estimated total time: 19h 7m 33s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 15s.
[2025-11-13 10:45:19,258][__main__][INFO] - Starting iteration 426.
[2025-11-13 10:45:19,260][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:45:19,261][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:28,176][__main__][INFO] - Number of regex retries in iteration 426: 0
[2025-11-13 10:45:28,177][__main__][INFO] - agents played in iteration 426 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:45:28,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:29,060][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:29,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:29,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:29,127][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:29,127][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:29,897][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:30,194][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:30,847][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:31,172][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:31,826][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:32,154][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:32,480][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:32,808][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:33,134][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:33,461][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:33,785][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:34,111][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:34,436][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:34,762][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:35,087][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:35,413][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:35,739][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:36,067][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:36,395][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:45:36,719][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:45:37,046][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:45:37,372][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:45:37,699][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:45:38,025][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:45:38,352][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:45:38,678][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:45:39,005][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:45:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:45:39,656][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:45:39,983][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:45:40,309][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:45:41,065][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:45:41,751][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:45:41,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:45:41,756][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:45:42,718][__main__][INFO] - Iteration 427 took 23s (38.01% Gen, 57.88% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 51m 3s. Estimated total time: 19h 32m 57s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 5s, 500 more iterations: 3h 15m 29s.
[2025-11-13 10:45:42,721][__main__][INFO] - Starting iteration 427.
[2025-11-13 10:45:42,724][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:45:42,724][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:51,957][__main__][INFO] - Number of regex retries in iteration 427: 0
[2025-11-13 10:45:51,958][__main__][INFO] - agents played in iteration 427 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:45:52,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:52,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:52,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:52,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:52,551][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:52,552][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:53,310][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:53,606][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:53,932][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:54,258][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:54,584][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:54,910][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:55,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:55,568][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:55,893][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:56,218][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:56,548][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:56,873][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:57,200][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:57,526][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:57,853][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:58,179][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:58,506][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:58,834][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:59,157][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:59,484][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:59,809][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:46:00,143][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:46:00,461][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:46:00,787][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:46:01,115][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:46:01,449][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:46:01,770][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:46:02,104][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:46:02,429][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:46:02,758][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:46:03,082][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:03,408][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:03,735][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:04,477][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:05,173][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:05,174][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:05,176][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:06,138][__main__][INFO] - Iteration 428 took 23s (39.43% Gen, 56.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 48m 28s. Estimated total time: 19h 30m 45s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 1s, 500 more iterations: 3h 15m 7s.
[2025-11-13 10:46:06,140][__main__][INFO] - Starting iteration 428.
[2025-11-13 10:46:06,143][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:46:06,144][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:46:15,078][__main__][INFO] - Number of regex retries in iteration 428: 0
[2025-11-13 10:46:15,079][__main__][INFO] - agents played in iteration 428 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:46:15,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:15,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:15,629][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:15,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:15,663][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:46:15,664][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:46:16,811][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:46:17,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:46:17,436][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:46:17,760][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:46:18,085][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:46:18,410][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:46:18,738][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:46:19,062][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:46:19,389][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:46:19,714][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:46:20,042][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:46:20,368][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:46:20,699][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:46:21,026][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:46:21,356][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:46:21,683][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:46:22,012][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:46:22,338][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:46:22,663][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:46:22,989][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:46:23,315][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:46:23,642][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:46:23,968][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:46:24,295][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:46:24,621][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:46:24,949][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:46:25,276][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:46:25,601][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:46:25,926][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:46:26,253][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:46:26,578][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:26,906][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:27,234][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:28,003][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:28,702][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:28,703][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:28,705][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:29,641][__main__][INFO] - Iteration 429 took 23s (38.02% Gen, 57.99% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 52m 16s. Estimated total time: 19h 34m 56s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 9s, 500 more iterations: 3h 15m 49s.
[2025-11-13 10:46:29,643][__main__][INFO] - Starting iteration 429.
[2025-11-13 10:46:29,647][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:46:29,647][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:46:38,561][__main__][INFO] - Number of regex retries in iteration 429: 0
[2025-11-13 10:46:38,562][__main__][INFO] - agents played in iteration 429 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:46:39,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:39,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:39,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:39,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:39,511][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:46:39,512][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:46:40,302][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:46:40,598][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:46:40,924][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:46:41,249][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:46:41,576][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:46:41,901][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:46:42,227][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:46:42,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:46:42,879][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:46:43,205][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:46:43,530][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:46:43,858][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:46:44,183][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:46:44,508][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:46:44,833][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:46:45,158][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:46:45,483][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:46:45,809][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:46:46,135][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:46:46,460][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:46:46,785][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:46:47,114][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:46:47,444][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:46:47,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:46:48,092][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:46:48,418][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:46:48,755][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:46:49,073][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:46:49,400][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:46:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:46:50,059][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:50,376][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:50,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:51,458][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:52,172][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:52,174][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:52,175][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:53,090][__main__][INFO] - Iteration 430 took 23s (38.02% Gen, 58.07% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 49m 9s. Estimated total time: 19h 32m 13s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 4s, 500 more iterations: 3h 15m 22s.
[2025-11-13 10:46:53,092][__main__][INFO] - Starting iteration 430.
[2025-11-13 10:46:53,095][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:46:53,097][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:02,655][__main__][INFO] - Number of regex retries in iteration 430: 0
[2025-11-13 10:47:02,656][__main__][INFO] - agents played in iteration 430 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:47:03,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:03,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:03,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:03,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:03,262][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:03,262][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:04,038][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:04,333][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:04,661][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:04,985][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:05,311][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:05,637][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:05,964][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:06,289][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:06,614][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:06,940][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:07,268][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:07,593][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:07,917][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:08,244][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:08,576][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:08,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:09,222][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:09,549][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:09,882][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:10,200][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:10,528][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:10,856][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:11,186][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:11,518][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:47:11,849][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:47:12,176][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:47:12,507][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:47:12,836][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:47:13,163][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:47:13,488][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:47:13,813][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:47:14,142][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:47:14,467][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:47:15,205][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:47:15,924][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:47:15,926][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:47:15,927][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:47:17,692][__main__][INFO] - Iteration 431 took 24s (38.86% Gen, 53.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 23s. Estimated total time: 20h 29m 52s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 59s, 500 more iterations: 3h 24m 58s.
[2025-11-13 10:47:17,693][__main__][INFO] - Starting iteration 431.
[2025-11-13 10:47:17,696][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:47:17,697][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:26,796][__main__][INFO] - Number of regex retries in iteration 431: 0
[2025-11-13 10:47:26,797][__main__][INFO] - agents played in iteration 431 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:47:27,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:27,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:27,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:27,384][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:27,385][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:27,385][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:28,159][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:28,457][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:28,784][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:29,109][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:29,436][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:29,761][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:30,087][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:30,412][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:30,738][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:31,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:31,390][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:31,717][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:32,043][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:32,369][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:32,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:33,020][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:33,348][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:33,675][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:34,004][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:34,329][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:34,654][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:34,979][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:35,305][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:35,634][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:47:35,961][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:47:36,294][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:47:36,615][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:47:36,941][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:47:37,267][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:47:37,593][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:47:37,918][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:47:38,245][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:47:38,573][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:47:39,327][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:47:40,038][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:47:40,039][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:47:40,041][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:47:41,094][__main__][INFO] - Iteration 432 took 23s (38.89% Gen, 56.61% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 46m 2s. Estimated total time: 19h 29m 55s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 59s, 500 more iterations: 3h 14m 59s.
[2025-11-13 10:47:41,097][__main__][INFO] - Starting iteration 432.
[2025-11-13 10:47:41,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:47:41,102][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:50,854][__main__][INFO] - Number of regex retries in iteration 432: 0
[2025-11-13 10:47:50,855][__main__][INFO] - agents played in iteration 432 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:47:51,344][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:51,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:51,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:51,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:51,445][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:51,446][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:52,214][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:52,511][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:52,838][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:53,163][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:53,491][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:53,817][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:54,143][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:54,469][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:54,794][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:55,119][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:55,445][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:56,102][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:56,428][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:56,754][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:57,080][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:57,405][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:57,731][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:58,056][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:58,383][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:58,707][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:59,032][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:59,359][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:59,689][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:48:00,015][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:48:00,341][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:48:00,667][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:48:00,993][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:48:01,320][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:48:01,645][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:48:01,971][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:02,298][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:02,626][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:03,383][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:04,095][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:04,097][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:04,099][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:05,104][__main__][INFO] - Iteration 433 took 24s (40.63% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 15m 54s. Estimated total time: 20h 0m 11s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 1s.
[2025-11-13 10:48:05,106][__main__][INFO] - Starting iteration 433.
[2025-11-13 10:48:05,108][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:05,109][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:48:14,349][__main__][INFO] - Number of regex retries in iteration 433: 0
[2025-11-13 10:48:14,350][__main__][INFO] - agents played in iteration 433 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:48:14,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:15,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:15,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:15,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:15,314][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:48:15,314][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:48:16,100][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:48:16,396][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:48:16,722][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:48:17,054][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:48:17,379][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:48:17,705][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:48:18,031][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:48:18,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:48:18,684][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:48:19,010][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:48:19,339][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:48:19,673][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:48:19,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:48:20,323][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:48:20,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:48:20,988][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:48:21,312][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:48:21,643][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:48:21,970][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:48:22,296][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:48:22,622][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:48:22,947][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:48:23,273][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:48:23,599][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:48:23,925][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:48:24,251][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:48:24,576][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:48:24,904][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:48:25,228][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:48:25,553][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:48:25,879][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:26,205][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:26,531][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:27,249][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:27,964][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:27,966][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:27,967][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:28,997][__main__][INFO] - Iteration 434 took 23s (38.68% Gen, 57.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 9m 47s. Estimated total time: 19h 54m 27s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 48s, 500 more iterations: 3h 19m 4s.
[2025-11-13 10:48:28,999][__main__][INFO] - Starting iteration 434.
[2025-11-13 10:48:29,002][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:29,002][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:48:38,266][__main__][INFO] - Number of regex retries in iteration 434: 0
[2025-11-13 10:48:38,267][__main__][INFO] - agents played in iteration 434 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:48:38,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:38,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:38,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:38,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:38,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:48:38,857][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:48:39,635][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:48:39,931][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:48:40,257][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:48:40,582][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:48:40,907][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:48:41,233][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:48:41,558][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:48:41,890][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:48:42,209][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:48:42,535][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:48:42,861][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:48:43,188][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:48:43,517][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:48:43,843][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:48:44,171][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:48:44,504][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:48:44,827][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:48:45,151][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:48:45,477][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:48:45,807][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:48:46,128][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:48:46,453][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:48:46,778][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:48:47,109][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:48:47,435][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:48:47,766][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:48:48,091][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:48:48,419][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:48:48,745][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:48:49,071][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:48:49,397][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:50,055][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:50,784][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:51,490][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:51,492][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:51,494][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:52,426][__main__][INFO] - Iteration 435 took 23s (39.55% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 46m 11s. Estimated total time: 19h 31m 14s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 2s, 500 more iterations: 3h 15m 12s.
[2025-11-13 10:48:52,428][__main__][INFO] - Starting iteration 435.
[2025-11-13 10:48:52,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:52,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:01,885][__main__][INFO] - Number of regex retries in iteration 435: 0
[2025-11-13 10:49:01,886][__main__][INFO] - agents played in iteration 435 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:49:02,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:02,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:02,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:02,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:02,474][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:02,475][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:03,258][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:03,554][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:03,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:04,207][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:04,532][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:04,857][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:05,188][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:05,508][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:05,833][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:06,159][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:06,491][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:06,815][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:07,142][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:07,468][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:07,803][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:08,122][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:08,447][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:08,773][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:09,106][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:09,425][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:09,752][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:10,079][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:10,412][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:10,734][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:11,061][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:11,387][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:11,713][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:12,040][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:12,365][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:12,691][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:49:13,343][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:49:13,670][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:49:14,410][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:49:15,112][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:49:15,113][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:49:15,114][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:49:16,048][__main__][INFO] - Iteration 436 took 23s (40.02% Gen, 56.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 55m 27s. Estimated total time: 19h 40m 55s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 49s.
[2025-11-13 10:49:16,050][__main__][INFO] - Starting iteration 436.
[2025-11-13 10:49:16,053][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:49:16,054][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:25,087][__main__][INFO] - Number of regex retries in iteration 436: 0
[2025-11-13 10:49:25,087][__main__][INFO] - agents played in iteration 436 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:49:25,584][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:25,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:25,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:25,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:25,686][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:25,686][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:26,467][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:26,765][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:27,091][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:27,417][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:27,742][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:28,067][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:28,393][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:28,719][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:29,045][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:29,371][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:29,700][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:30,027][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:30,353][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:30,678][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:31,003][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:31,329][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:31,656][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:31,980][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:32,307][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:32,632][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:32,957][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:33,285][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:33,610][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:33,935][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:34,263][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:34,590][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:34,916][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:35,242][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:35,570][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:35,898][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:36,224][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:49:36,549][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:49:36,874][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:49:37,616][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:49:38,340][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:49:38,341][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:49:38,342][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:49:39,460][__main__][INFO] - Iteration 437 took 23s (38.59% Gen, 56.63% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 44m 33s. Estimated total time: 19h 30m 23s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 0s, 500 more iterations: 3h 15m 3s.
[2025-11-13 10:49:39,462][__main__][INFO] - Starting iteration 437.
[2025-11-13 10:49:39,466][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:49:39,466][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:48,553][__main__][INFO] - Number of regex retries in iteration 437: 0
[2025-11-13 10:49:48,554][__main__][INFO] - agents played in iteration 437 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:49:49,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:49,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:49,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:49,158][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:49,158][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:49,159][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:49,917][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:50,214][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:50,540][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:50,867][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:51,193][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:51,524][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:51,851][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:52,177][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:52,503][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:52,829][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:53,155][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:53,485][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:53,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:54,138][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:54,463][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:54,789][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:55,115][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:55,446][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:55,779][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:56,104][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:56,430][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:56,757][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:57,085][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:57,416][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:57,743][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:58,070][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:58,395][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:58,722][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:59,049][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:59,375][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:59,701][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:50:00,028][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:50:00,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:01,122][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:01,883][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:01,885][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:01,886][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:02,796][__main__][INFO] - Iteration 438 took 23s (38.95% Gen, 57.14% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 40m 20s. Estimated total time: 19h 26m 34s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 25s.
[2025-11-13 10:50:02,799][__main__][INFO] - Starting iteration 438.
[2025-11-13 10:50:02,802][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:50:02,802][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:11,633][__main__][INFO] - Number of regex retries in iteration 438: 0
[2025-11-13 10:50:11,633][__main__][INFO] - agents played in iteration 438 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:50:12,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:12,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:12,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:12,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:12,238][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:50:12,239][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:50:13,019][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:50:13,317][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:50:13,645][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:50:13,968][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:50:14,294][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:50:14,619][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:50:14,947][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:50:15,270][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:50:15,594][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:50:15,921][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:50:16,248][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:50:16,575][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:50:16,900][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:50:17,226][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:50:17,552][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:50:17,878][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:50:18,203][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:50:18,528][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:50:18,854][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:50:19,180][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:50:19,505][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:50:19,832][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:50:20,159][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:50:20,484][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:50:20,815][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:50:21,144][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:50:21,471][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:50:21,795][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:50:22,128][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:50:22,453][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:50:22,780][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:50:23,106][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:50:23,433][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:24,181][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:24,943][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:24,944][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:24,946][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:25,923][__main__][INFO] - Iteration 439 took 23s (38.19% Gen, 57.57% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 29m 30s. Estimated total time: 19h 16m 7s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 32s, 500 more iterations: 3h 12m 41s.
[2025-11-13 10:50:25,925][__main__][INFO] - Starting iteration 439.
[2025-11-13 10:50:25,929][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:50:25,929][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:35,471][__main__][INFO] - Number of regex retries in iteration 439: 0
[2025-11-13 10:50:35,471][__main__][INFO] - agents played in iteration 439 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:50:35,969][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,072][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:50:36,072][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:50:36,879][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:50:37,178][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:50:37,506][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:50:37,833][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:50:38,159][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:50:38,485][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:50:38,812][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:50:39,137][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:50:39,464][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:50:39,791][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:50:40,115][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:50:40,441][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:50:40,768][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:50:41,095][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:50:41,421][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:50:41,747][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:50:42,077][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:50:42,404][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:50:42,736][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:50:43,062][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:50:43,390][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:50:43,716][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:50:44,047][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:50:44,375][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:50:44,706][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:50:45,034][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:50:45,362][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:50:45,688][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:50:46,013][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:50:46,344][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:50:46,674][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:50:47,004][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:50:47,330][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:48,055][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:48,796][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:48,798][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:48,799][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:49,862][__main__][INFO] - Iteration 440 took 23s (39.87% Gen, 55.69% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 9m 40s. Estimated total time: 19h 56m 41s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 26s.
[2025-11-13 10:50:49,864][__main__][INFO] - Starting iteration 440.
[2025-11-13 10:50:49,867][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:50:49,867][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:59,690][__main__][INFO] - Number of regex retries in iteration 440: 0
[2025-11-13 10:50:59,690][__main__][INFO] - agents played in iteration 440 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:51:00,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,262][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,296][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:00,297][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:01,092][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:01,388][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:01,715][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:02,366][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:02,692][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:03,016][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:03,342][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:03,667][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:03,992][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:04,317][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:04,643][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:04,968][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:05,293][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:05,619][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:05,948][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:06,275][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:06,601][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:06,926][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:07,256][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:07,584][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:07,917][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:08,244][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:08,570][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:08,897][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:09,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:09,559][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:09,886][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:10,212][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:10,543][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:10,871][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:11,197][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:11,527][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:51:12,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:51:13,063][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:51:13,066][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:51:13,068][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:51:15,000][__main__][INFO] - Iteration 441 took 25s (39.08% Gen, 53.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 9m 14s. Estimated total time: 20h 56m 40s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 53s, 500 more iterations: 3h 29m 26s.
[2025-11-13 10:51:15,002][__main__][INFO] - Starting iteration 441.
[2025-11-13 10:51:15,006][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:51:15,007][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:51:24,485][__main__][INFO] - Number of regex retries in iteration 441: 0
[2025-11-13 10:51:24,486][__main__][INFO] - agents played in iteration 441 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:51:24,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:25,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:25,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:25,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:25,098][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:25,099][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:25,861][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:26,157][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:26,484][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:26,810][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:27,135][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:27,789][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:28,114][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:28,440][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:28,765][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:29,091][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:29,419][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:29,745][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:30,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:30,397][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:30,725][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:31,049][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:31,376][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:31,702][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:32,027][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:32,353][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:32,679][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:33,006][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:33,662][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:33,988][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:34,314][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:34,638][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:34,964][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:35,289][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:35,614][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:35,940][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:36,268][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:51:37,011][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:51:37,760][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:51:37,762][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:51:37,764][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:51:38,718][__main__][INFO] - Iteration 442 took 23s (39.97% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 57m 50s. Estimated total time: 19h 45m 40s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 36s.
[2025-11-13 10:51:38,720][__main__][INFO] - Starting iteration 442.
[2025-11-13 10:51:38,723][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:51:38,723][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:51:48,139][__main__][INFO] - Number of regex retries in iteration 442: 0
[2025-11-13 10:51:48,139][__main__][INFO] - agents played in iteration 442 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:51:48,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:48,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:48,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:48,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:48,751][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:48,751][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:49,519][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:49,817][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:50,142][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:50,467][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:50,792][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:51,117][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:51,443][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:51,771][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:52,097][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:52,428][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:52,753][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:53,080][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:53,405][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:53,735][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:54,062][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:54,388][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:54,717][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:55,042][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:55,699][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:56,026][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:56,351][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:56,677][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:57,001][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:57,330][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:57,657][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:57,983][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:58,307][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:58,632][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:58,957][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:59,283][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:59,610][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:59,939][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:52:00,674][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:52:01,399][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:52:01,401][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:52:01,403][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:02,416][__main__][INFO] - Iteration 443 took 23s (39.74% Gen, 55.98% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 29s. Estimated total time: 19h 44m 43s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 29s, 500 more iterations: 3h 17m 27s.
[2025-11-13 10:52:02,418][__main__][INFO] - Starting iteration 443.
[2025-11-13 10:52:02,421][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:02,422][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:11,548][__main__][INFO] - Number of regex retries in iteration 443: 0
[2025-11-13 10:52:11,548][__main__][INFO] - agents played in iteration 443 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:52:12,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:12,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:12,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:12,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:12,156][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:12,157][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:52:12,940][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:52:13,237][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:52:13,564][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:52:13,889][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:52:14,215][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:52:14,542][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:52:14,868][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:52:15,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:52:15,518][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:52:15,844][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:52:16,170][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:52:16,495][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:52:16,820][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:52:17,145][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:52:17,469][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:52:17,796][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:52:18,121][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:52:18,447][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:52:18,771][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:52:19,097][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:52:19,422][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:52:19,746][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:52:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:52:20,396][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:52:20,721][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:52:21,046][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:52:21,370][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:52:21,694][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:52:22,020][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:52:22,345][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:52:22,671][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:52:22,997][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:52:23,323][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:52:24,070][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:52:24,807][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:52:24,808][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:52:24,810][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:25,827][__main__][INFO] - Iteration 444 took 23s (38.99% Gen, 56.66% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 41m 42s. Estimated total time: 19h 30m 19s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 0s, 500 more iterations: 3h 15m 3s.
[2025-11-13 10:52:25,829][__main__][INFO] - Starting iteration 444.
[2025-11-13 10:52:25,832][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:25,832][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:35,479][__main__][INFO] - Number of regex retries in iteration 444: 0
[2025-11-13 10:52:35,480][__main__][INFO] - agents played in iteration 444 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:52:35,988][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:36,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:36,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:36,090][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:36,090][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:36,090][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:52:36,862][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:52:37,159][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:52:37,486][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:52:37,812][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:52:38,138][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:52:38,466][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:52:38,794][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:52:39,121][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:52:39,447][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:52:39,773][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:52:40,098][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:52:40,425][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:52:40,749][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:52:41,076][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:52:41,406][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:52:41,732][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:52:42,058][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:52:42,382][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:52:42,709][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:52:43,036][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:52:43,361][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:52:43,685][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:52:44,012][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:52:44,341][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:52:44,669][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:52:44,995][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:52:45,320][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:52:45,648][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:52:45,973][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:52:46,303][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:52:46,628][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:52:46,953][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:52:47,281][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:52:48,017][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:52:48,744][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:52:48,746][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:52:48,748][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:49,737][__main__][INFO] - Iteration 445 took 23s (40.35% Gen, 55.50% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 6m 17s. Estimated total time: 19h 55m 18s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 13s.
[2025-11-13 10:52:49,739][__main__][INFO] - Starting iteration 445.
[2025-11-13 10:52:49,742][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:49,742][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:59,109][__main__][INFO] - Number of regex retries in iteration 445: 0
[2025-11-13 10:52:59,109][__main__][INFO] - agents played in iteration 445 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:52:59,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:59,657][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:59,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:59,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:59,726][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:59,726][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:53:00,482][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:53:00,778][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:53:01,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:53:01,430][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:53:01,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:02,085][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:02,411][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:02,738][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:03,062][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:03,389][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:03,713][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:04,039][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:04,364][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:04,690][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:05,019][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:05,348][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:05,672][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:05,999][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:06,329][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:06,660][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:06,988][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:07,315][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:07,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:07,976][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:08,307][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:08,633][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:08,965][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:09,294][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:09,622][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:09,947][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:10,273][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:10,599][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:10,927][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:11,689][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:12,383][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:12,387][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:12,388][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:13,497][__main__][INFO] - Iteration 446 took 23s (39.43% Gen, 55.90% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 58m 23s. Estimated total time: 19h 47m 47s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 57s.
[2025-11-13 10:53:13,499][__main__][INFO] - Starting iteration 446.
[2025-11-13 10:53:13,501][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:13,502][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:53:22,284][__main__][INFO] - Number of regex retries in iteration 446: 0
[2025-11-13 10:53:22,285][__main__][INFO] - agents played in iteration 446 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:53:22,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:22,810][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:22,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:22,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:22,878][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:53:22,879][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:53:23,666][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:53:23,963][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:53:24,290][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:53:24,616][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:53:24,942][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:25,268][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:25,592][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:25,918][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:26,244][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:26,570][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:26,894][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:27,220][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:27,544][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:27,871][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:28,196][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:28,522][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:28,858][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:29,185][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:29,509][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:29,837][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:30,170][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:30,497][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:30,822][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:31,149][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:31,476][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:31,804][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:32,134][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:32,464][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:32,787][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:33,116][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:33,442][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:33,777][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:34,097][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:34,823][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:35,570][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:35,575][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:35,577][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:36,739][__main__][INFO] - Iteration 447 took 23s (37.79% Gen, 57.20% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 32m 7s. Estimated total time: 19h 21m 55s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 39s.
[2025-11-13 10:53:36,741][__main__][INFO] - Starting iteration 447.
[2025-11-13 10:53:36,744][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:36,744][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:53:45,721][__main__][INFO] - Number of regex retries in iteration 447: 0
[2025-11-13 10:53:45,721][__main__][INFO] - agents played in iteration 447 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:53:46,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:46,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:46,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:46,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:46,319][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:53:46,320][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:53:47,102][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:53:47,398][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:53:47,725][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:53:48,051][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:53:48,386][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:48,703][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:49,029][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:49,356][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:49,692][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:50,014][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:50,342][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:50,669][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:50,998][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:51,324][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:51,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:51,975][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:52,301][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:52,627][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:52,952][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:53,279][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:53,605][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:53,930][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:54,255][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:54,580][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:54,908][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:55,559][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:55,888][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:56,212][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:56,538][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:56,864][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:57,191][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:57,514][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:58,259][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:59,012][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:59,014][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:59,015][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:59,931][__main__][INFO] - Iteration 448 took 23s (38.71% Gen, 57.33% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 29m 13s. Estimated total time: 19h 19m 24s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 10:53:59,933][__main__][INFO] - Starting iteration 448.
[2025-11-13 10:53:59,936][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:59,936][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:09,695][__main__][INFO] - Number of regex retries in iteration 448: 0
[2025-11-13 10:54:09,695][__main__][INFO] - agents played in iteration 448 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:54:10,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:10,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:10,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:10,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:10,283][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:10,284][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:11,056][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:11,353][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:54:11,678][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:54:12,012][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:54:12,332][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:54:12,658][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:54:12,984][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:54:13,312][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:54:13,636][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:54:13,961][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:54:14,288][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:54:14,616][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:54:14,942][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:54:15,267][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:54:15,592][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:54:15,920][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:54:16,250][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:54:16,575][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:54:16,902][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:54:17,230][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:54:17,557][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:54:17,884][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:54:18,209][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:54:18,536][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:54:18,865][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:54:19,191][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:54:19,518][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:54:19,843][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:54:20,172][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:54:20,503][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:54:20,827][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:54:21,154][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:54:21,478][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:54:22,236][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:54:22,987][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:54:22,989][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:54:22,991][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:54:24,024][__main__][INFO] - Iteration 449 took 24s (40.51% Gen, 55.19% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 13m 54s. Estimated total time: 20h 4m 29s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 8s, 500 more iterations: 3h 20m 44s.
[2025-11-13 10:54:24,028][__main__][INFO] - Starting iteration 449.
[2025-11-13 10:54:24,030][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:54:24,031][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:33,697][__main__][INFO] - Number of regex retries in iteration 449: 0
[2025-11-13 10:54:33,697][__main__][INFO] - agents played in iteration 449 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:54:34,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:34,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:34,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:34,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:34,302][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:34,303][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:35,060][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:35,357][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:54:35,684][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:54:36,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:54:36,336][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:54:36,661][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:54:36,987][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:54:37,312][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:54:37,638][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:54:37,963][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:54:38,288][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:54:38,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:54:38,940][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:54:39,265][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:54:39,589][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:54:39,918][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:54:40,244][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:54:40,572][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:54:40,899][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:54:41,230][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:54:41,557][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:54:41,884][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:54:42,210][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:54:42,535][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:54:42,859][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:54:43,185][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:54:43,515][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:54:43,845][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:54:44,169][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:54:44,494][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:54:44,821][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:54:45,149][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:54:45,475][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:54:46,229][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:54:46,965][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:54:46,966][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:54:46,968][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:54:48,042][__main__][INFO] - Iteration 450 took 24s (40.26% Gen, 55.27% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 9m 38s. Estimated total time: 20h 0m 37s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 1s, 500 more iterations: 3h 20m 6s.
[2025-11-13 10:54:48,044][__main__][INFO] - Starting iteration 450.
[2025-11-13 10:54:48,049][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:54:48,049][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:58,069][__main__][INFO] - Number of regex retries in iteration 450: 0
[2025-11-13 10:54:58,070][__main__][INFO] - agents played in iteration 450 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:54:58,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:58,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:58,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:58,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:58,664][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:58,664][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:59,427][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:59,724][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:55:00,052][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:55:00,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:55:00,714][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:01,039][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:01,364][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:01,688][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:02,015][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:02,341][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:02,666][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:02,992][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:03,316][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:03,644][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:03,972][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:04,298][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:04,622][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:04,948][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:05,273][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:05,599][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:05,925][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:06,250][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:06,576][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:06,902][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:07,227][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:07,552][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:07,876][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:08,201][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:08,525][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:08,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:09,174][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:09,499][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:09,825][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:10,559][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:55:11,314][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:55:11,315][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:55:11,317][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:55:13,263][__main__][INFO] - Iteration 451 took 25s (39.74% Gen, 52.53% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 9m 23s. Estimated total time: 21h 0m 47s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 1s, 500 more iterations: 3h 30m 7s.
[2025-11-13 10:55:13,265][__main__][INFO] - Starting iteration 451.
[2025-11-13 10:55:13,267][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:55:13,268][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:55:23,480][__main__][INFO] - Number of regex retries in iteration 451: 0
[2025-11-13 10:55:23,481][__main__][INFO] - agents played in iteration 451 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:55:23,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:24,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:24,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:24,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:24,081][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:55:24,081][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:55:24,853][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:55:25,148][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:55:25,475][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:55:25,807][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:55:26,136][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:26,462][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:26,789][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:27,119][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:27,444][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:27,769][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:28,098][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:28,425][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:28,751][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:29,077][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:29,408][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:29,733][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:30,058][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:30,382][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:30,707][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:31,033][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:31,357][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:31,681][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:32,007][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:32,334][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:32,658][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:32,985][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:33,311][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:33,637][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:33,965][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:34,293][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:34,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:34,948][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:35,273][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:36,048][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:55:36,755][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:55:36,756][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:55:36,758][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:55:37,687][__main__][INFO] - Iteration 452 took 24s (41.82% Gen, 54.37% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 29m 12s. Estimated total time: 20h 21m 1s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 30s.
[2025-11-13 10:55:37,689][__main__][INFO] - Starting iteration 452.
[2025-11-13 10:55:37,691][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:55:37,692][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:55:47,318][__main__][INFO] - Number of regex retries in iteration 452: 0
[2025-11-13 10:55:47,319][__main__][INFO] - agents played in iteration 452 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:55:47,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:47,848][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:47,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:47,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:47,935][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:55:47,935][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:55:48,703][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:55:49,999][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:55:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:55:49,652][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:55:49,977][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:50,302][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:50,627][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:50,951][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:51,276][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:51,600][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:51,925][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:52,251][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:52,577][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:52,904][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:53,231][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:53,558][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:53,884][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:54,211][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:54,537][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:54,864][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:55,189][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:55,515][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:55,840][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:56,165][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:56,492][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:56,818][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:57,146][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:57,474][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:57,802][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:58,128][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:58,453][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:58,780][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:59,106][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:59,870][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:56:00,563][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:56:00,564][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:56:00,568][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:56:01,502][__main__][INFO] - Iteration 453 took 23s (40.43% Gen, 55.64% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 58m 21s. Estimated total time: 19h 50m 34s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 25s.
[2025-11-13 10:56:01,504][__main__][INFO] - Starting iteration 453.
[2025-11-13 10:56:01,507][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:56:01,507][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:11,661][__main__][INFO] - Number of regex retries in iteration 453: 0
[2025-11-13 10:56:11,661][__main__][INFO] - agents played in iteration 453 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:56:12,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:12,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:12,209][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:12,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:12,243][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:12,244][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:56:13,007][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:56:13,303][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:56:13,631][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:56:13,959][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:56:14,284][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:56:14,610][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:56:14,940][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:56:15,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:56:15,595][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:56:15,921][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:56:16,250][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:56:16,579][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:56:16,909][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:56:17,236][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:56:17,563][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:56:17,888][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:56:18,212][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:56:18,537][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:56:18,862][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:56:19,191][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:56:19,519][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:56:19,847][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:56:20,173][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:56:20,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:56:20,825][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:56:21,150][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:56:21,475][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:56:21,803][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:56:22,128][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:56:22,455][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:56:22,780][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:56:23,105][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:56:23,432][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:56:24,191][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:56:24,901][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:56:24,903][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:56:24,905][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:56:25,900][__main__][INFO] - Iteration 454 took 24s (41.62% Gen, 54.29% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 27m 6s. Estimated total time: 20h 19m 43s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 39s, 500 more iterations: 3h 23m 17s.
[2025-11-13 10:56:25,902][__main__][INFO] - Starting iteration 454.
[2025-11-13 10:56:25,905][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:56:25,906][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:35,240][__main__][INFO] - Number of regex retries in iteration 454: 0
[2025-11-13 10:56:35,240][__main__][INFO] - agents played in iteration 454 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:56:35,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:35,758][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:35,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:35,825][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:35,826][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:35,826][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:56:36,598][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:56:36,897][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:56:37,224][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:56:37,554][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:56:37,879][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:56:38,205][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:56:38,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:56:38,855][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:56:39,182][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:56:39,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:56:39,836][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:56:40,164][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:56:40,490][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:56:40,815][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:56:41,141][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:56:41,467][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:56:41,792][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:56:42,124][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:56:42,448][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:56:42,772][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:56:43,097][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:56:43,428][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:56:43,752][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:56:44,077][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:56:44,404][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:56:44,727][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:56:45,053][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:56:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:56:45,705][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:56:46,033][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:56:46,363][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:56:46,691][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:56:47,023][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:56:47,767][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:56:48,502][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:56:48,503][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:56:48,505][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:56:49,410][__main__][INFO] - Iteration 455 took 23s (39.71% Gen, 56.43% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 42m 15s. Estimated total time: 19h 35m 16s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 52s.
[2025-11-13 10:56:49,412][__main__][INFO] - Starting iteration 455.
[2025-11-13 10:56:49,415][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:56:49,415][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:59,267][__main__][INFO] - Number of regex retries in iteration 455: 0
[2025-11-13 10:56:59,268][__main__][INFO] - agents played in iteration 455 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:56:59,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:59,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:59,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:59,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:59,856][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:59,857][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:57:00,615][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:57:00,913][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:57:01,239][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:01,571][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:01,896][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:02,224][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:02,552][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:02,880][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:03,212][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:03,868][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:04,195][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:04,527][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:04,858][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:05,189][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:05,519][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:05,847][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:06,177][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:06,504][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:06,831][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:07,161][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:07,488][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:07,815][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:08,145][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:08,467][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:08,797][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:09,124][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:09,452][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:09,779][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:10,106][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:10,437][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:10,768][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:11,098][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:57:11,846][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:57:12,567][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:57:12,569][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:57:12,570][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:57:13,524][__main__][INFO] - Iteration 456 took 24s (40.87% Gen, 55.17% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 12m 5s. Estimated total time: 20h 5m 30s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 11s, 500 more iterations: 3h 20m 55s.
[2025-11-13 10:57:13,527][__main__][INFO] - Starting iteration 456.
[2025-11-13 10:57:13,530][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:57:13,531][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:57:23,783][__main__][INFO] - Number of regex retries in iteration 456: 0
[2025-11-13 10:57:23,784][__main__][INFO] - agents played in iteration 456 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:57:24,272][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:24,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:24,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:24,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:24,375][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:57:24,376][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:57:25,126][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:57:25,423][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:57:25,747][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:26,075][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:26,403][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:26,734][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:27,061][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:27,385][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:27,716][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:28,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:28,374][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:28,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:29,032][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:29,358][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:29,682][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:30,008][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:30,339][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:30,662][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:30,988][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:31,313][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:31,641][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:31,967][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:32,626][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:32,951][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:33,276][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:33,601][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:33,928][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:34,262][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:34,581][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:34,906][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:35,231][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:35,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:57:36,298][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:57:37,014][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:57:37,015][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:57:37,017][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:57:38,112][__main__][INFO] - Iteration 457 took 24s (41.70% Gen, 53.83% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 35m 19s. Estimated total time: 20h 29m 8s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 58s, 500 more iterations: 3h 24m 51s.
[2025-11-13 10:57:38,114][__main__][INFO] - Starting iteration 457.
[2025-11-13 10:57:38,117][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:57:38,118][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:57:47,756][__main__][INFO] - Number of regex retries in iteration 457: 0
[2025-11-13 10:57:47,757][__main__][INFO] - agents played in iteration 457 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:57:48,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:48,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:48,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:48,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:48,338][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:57:48,338][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:57:49,101][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:57:49,398][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:57:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:50,058][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:50,376][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:50,701][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:51,025][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:51,357][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:51,678][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:52,004][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:52,333][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:52,664][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:52,984][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:53,309][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:53,634][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:53,967][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:54,287][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:54,611][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:54,936][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:55,263][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:55,590][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:55,918][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:56,246][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:56,571][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:56,897][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:57,222][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:57,549][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:57,876][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:58,202][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:58,527][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:58,852][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:59,178][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:59,505][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:58:00,233][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:58:00,945][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:58:00,947][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:58:00,949][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:02,054][__main__][INFO] - Iteration 458 took 23s (40.27% Gen, 55.11% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 2m 40s. Estimated total time: 19h 56m 53s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 28s.
[2025-11-13 10:58:02,056][__main__][INFO] - Starting iteration 458.
[2025-11-13 10:58:02,060][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:58:02,060][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:58:12,865][__main__][INFO] - Number of regex retries in iteration 458: 0
[2025-11-13 10:58:12,866][__main__][INFO] - agents played in iteration 458 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:58:13,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:13,379][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:13,412][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:13,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:13,446][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:58:13,446][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:58:14,238][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:58:14,535][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:58:14,864][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:58:15,191][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:58:15,522][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:58:15,849][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:58:16,174][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:58:16,499][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:58:16,825][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:58:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:58:17,475][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:58:17,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:58:18,126][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:58:18,453][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:58:18,777][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:58:19,102][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:58:19,428][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:58:19,753][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:58:20,078][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:58:20,403][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:58:20,740][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:58:21,065][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:58:21,390][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:58:21,714][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:58:22,041][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:58:22,369][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:58:22,694][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:58:23,019][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:58:23,346][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:58:23,671][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:58:23,997][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:58:24,324][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:58:24,656][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:58:25,392][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:58:26,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:58:26,107][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:58:26,109][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:27,029][__main__][INFO] - Iteration 459 took 24s (43.27% Gen, 53.04% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 53m 53s. Estimated total time: 20h 48m 32s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 37s, 500 more iterations: 3h 28m 5s.
[2025-11-13 10:58:27,031][__main__][INFO] - Starting iteration 459.
[2025-11-13 10:58:27,034][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:58:27,035][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:58:37,262][__main__][INFO] - Number of regex retries in iteration 459: 0
[2025-11-13 10:58:37,263][__main__][INFO] - agents played in iteration 459 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:58:37,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:37,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:37,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:37,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:37,864][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:58:37,864][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:58:38,632][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:58:38,930][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:58:39,256][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:58:39,582][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:58:39,907][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:58:40,234][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:58:40,559][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:58:40,884][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:58:41,210][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:58:41,536][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:58:41,860][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:58:42,185][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:58:42,512][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:58:42,835][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:58:43,158][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:58:43,483][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:58:43,808][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:58:44,133][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:58:44,464][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:58:44,790][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:58:45,114][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:58:45,439][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:58:45,768][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:58:46,094][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:58:46,417][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:58:46,744][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:58:47,072][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:58:47,398][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:58:47,725][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:58:48,053][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:58:48,380][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:58:48,709][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:58:49,037][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:58:49,797][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:58:50,486][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:58:50,489][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:58:50,491][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:51,439][__main__][INFO] - Iteration 460 took 24s (41.90% Gen, 54.20% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 25m 16s. Estimated total time: 20h 20m 18s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 40s, 500 more iterations: 3h 23m 23s.
[2025-11-13 10:58:51,442][__main__][INFO] - Starting iteration 460.
[2025-11-13 10:58:51,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:58:51,445][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:59:02,204][__main__][INFO] - Number of regex retries in iteration 460: 0
[2025-11-13 10:59:02,205][__main__][INFO] - agents played in iteration 460 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:59:02,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:02,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:02,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:02,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:02,788][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:59:02,788][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:03,530][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:03,827][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:04,168][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:04,496][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:04,824][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:05,150][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:05,480][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:05,804][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:06,130][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:06,455][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:06,784][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:07,109][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:07,433][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:07,758][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:08,084][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:08,414][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:08,738][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:09,066][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:09,387][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:59:09,712][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:59:10,038][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:59:10,366][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:59:10,690][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:59:11,015][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:59:11,341][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:59:11,679][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:59:11,997][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:59:12,325][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:59:12,656][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:59:12,989][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:59:13,311][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:59:13,639][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:59:13,965][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:59:14,709][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:59:15,405][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:59:15,407][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:59:15,408][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:59:17,540][__main__][INFO] - Iteration 461 took 26s (41.23% Gen, 50.59% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 49m 21s. Estimated total time: 21h 44m 50s. Time estimates for 10 more iterations: 4m 20s, 100 more iterations: 43m 29s, 500 more iterations: 3h 37m 28s.
[2025-11-13 10:59:17,542][__main__][INFO] - Starting iteration 461.
[2025-11-13 10:59:17,545][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:59:17,546][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:59:27,829][__main__][INFO] - Number of regex retries in iteration 461: 0
[2025-11-13 10:59:27,830][__main__][INFO] - agents played in iteration 461 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:59:28,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:28,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:28,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:28,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:28,420][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:59:28,420][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:29,213][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:29,510][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:29,838][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:30,169][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:30,491][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:30,817][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:31,142][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:31,467][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:31,797][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:32,121][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:32,446][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:32,775][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:33,096][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:33,424][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:33,749][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:34,080][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:34,725][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:35,051][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:59:35,381][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:59:35,701][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:59:36,028][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:59:36,353][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:59:36,685][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:59:37,006][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:59:37,331][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:59:37,659][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:59:37,993][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:59:38,315][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:59:38,644][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:59:38,970][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:59:39,301][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:59:39,623][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:59:40,410][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:59:41,150][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:59:41,152][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:59:41,153][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:59:42,203][__main__][INFO] - Iteration 462 took 24s (41.71% Gen, 54.03% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 37m 2s. Estimated total time: 20h 32m 55s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 5s, 500 more iterations: 3h 25m 29s.
[2025-11-13 10:59:42,205][__main__][INFO] - Starting iteration 462.
[2025-11-13 10:59:42,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:59:42,208][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:59:52,471][__main__][INFO] - Number of regex retries in iteration 462: 0
[2025-11-13 10:59:52,472][__main__][INFO] - agents played in iteration 462 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 10:59:52,952][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:52,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:53,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:53,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:53,056][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:59:53,056][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:53,847][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:54,144][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:54,470][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:54,794][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:55,119][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:55,444][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:55,768][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:56,093][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:56,417][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:56,741][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:57,069][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:57,394][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:57,721][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:58,050][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:58,375][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:58,702][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:59,026][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:59,351][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:59,677][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:00:00,006][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:00:00,332][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:00:00,656][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:00:00,981][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:00:01,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:00:01,632][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:00:01,964][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:00:02,291][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:00:02,617][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:00:02,941][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:00:03,279][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:00:03,606][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:00:03,931][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:00:04,257][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:00:05,039][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:00:05,789][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:00:05,790][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:00:05,792][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:00:06,987][__main__][INFO] - Iteration 463 took 24s (41.42% Gen, 53.75% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 42m 41s. Estimated total time: 20h 38m 59s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 17s, 500 more iterations: 3h 26m 29s.
[2025-11-13 11:00:06,989][__main__][INFO] - Starting iteration 463.
[2025-11-13 11:00:06,993][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:00:06,994][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:00:16,631][__main__][INFO] - Number of regex retries in iteration 463: 0
[2025-11-13 11:00:16,632][__main__][INFO] - agents played in iteration 463 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:00:17,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:17,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:17,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:17,225][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:17,226][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:00:17,226][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:00:18,020][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:00:18,315][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:00:18,641][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:00:18,971][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:00:19,296][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:00:19,620][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:00:19,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:00:20,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:00:20,602][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:00:20,926][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:00:21,254][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:00:21,584][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:00:21,907][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:00:22,231][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:00:22,557][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:00:22,895][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:00:23,217][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:00:23,542][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:00:23,868][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:00:24,201][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:00:24,522][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:00:24,847][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:00:25,172][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:00:25,498][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:00:25,823][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:00:26,150][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:00:26,476][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:00:26,806][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:00:27,126][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:00:27,453][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:00:27,784][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:00:28,115][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:00:28,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:00:29,198][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:00:29,954][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:00:29,956][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:00:29,957][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:00:31,039][__main__][INFO] - Iteration 464 took 24s (40.08% Gen, 55.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 5m 40s. Estimated total time: 20h 2m 22s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 4s, 500 more iterations: 3h 20m 23s.
[2025-11-13 11:00:31,042][__main__][INFO] - Starting iteration 464.
[2025-11-13 11:00:31,045][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:00:31,046][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:00:41,539][__main__][INFO] - Number of regex retries in iteration 464: 0
[2025-11-13 11:00:41,540][__main__][INFO] - agents played in iteration 464 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:00:42,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:42,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:42,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:42,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:42,132][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:00:42,133][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:00:42,913][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:00:43,209][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:00:43,535][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:00:43,861][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:00:44,194][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:00:44,521][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:00:44,847][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:00:45,174][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:00:45,501][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:00:45,830][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:00:46,155][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:00:46,480][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:00:46,805][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:00:47,133][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:00:47,458][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:00:47,784][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:00:48,108][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:00:48,433][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:00:48,759][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:00:49,086][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:00:49,413][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:00:49,738][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:00:50,064][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:00:50,395][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:00:50,715][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:00:51,044][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:00:51,371][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:00:51,705][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:00:52,025][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:00:52,349][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:00:52,677][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:00:53,007][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:00:53,326][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:00:54,079][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:00:54,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:00:54,839][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:00:54,841][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:00:55,852][__main__][INFO] - Iteration 465 took 24s (42.30% Gen, 53.62% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 43m 14s. Estimated total time: 20h 40m 22s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 20s, 500 more iterations: 3h 26m 43s.
[2025-11-13 11:00:55,854][__main__][INFO] - Starting iteration 465.
[2025-11-13 11:00:55,857][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:00:55,858][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:01:05,582][__main__][INFO] - Number of regex retries in iteration 465: 0
[2025-11-13 11:01:05,583][__main__][INFO] - agents played in iteration 465 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:01:06,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:06,089][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:06,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:06,159][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:06,160][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:01:06,160][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:01:06,958][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:01:07,256][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:01:07,586][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:01:07,916][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:01:08,243][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:01:08,569][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:01:08,901][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:01:09,225][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:01:09,551][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:01:09,876][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:01:10,201][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:01:10,526][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:01:10,851][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:01:11,176][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:01:11,502][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:01:11,827][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:01:12,152][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:01:12,477][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:01:12,804][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:01:13,131][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:01:13,455][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:01:13,781][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:01:14,111][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:01:14,439][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:01:14,768][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:01:15,097][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:01:15,424][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:01:15,748][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:01:16,076][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:01:16,400][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:01:16,726][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:01:17,054][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:01:17,381][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:01:18,198][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:01:18,948][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:01:18,950][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:01:18,952][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:01:20,034][__main__][INFO] - Iteration 466 took 24s (40.22% Gen, 55.29% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 11m 22s. Estimated total time: 20h 8m 53s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 17s, 500 more iterations: 3h 21m 28s.
[2025-11-13 11:01:20,036][__main__][INFO] - Starting iteration 466.
[2025-11-13 11:01:20,039][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:01:20,040][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:01:29,639][__main__][INFO] - Number of regex retries in iteration 466: 0
[2025-11-13 11:01:29,640][__main__][INFO] - agents played in iteration 466 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:01:30,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:30,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:30,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:30,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:30,253][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:01:30,253][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:01:31,035][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:01:31,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:01:31,658][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:01:31,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:01:32,316][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:01:32,636][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:01:32,963][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:01:33,289][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:01:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:01:33,941][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:01:34,267][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:01:34,593][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:01:34,921][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:01:35,246][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:01:35,571][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:01:35,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:01:36,222][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:01:36,550][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:01:36,876][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:01:37,201][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:01:37,529][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:01:37,855][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:01:38,186][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:01:38,512][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:01:38,838][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:01:39,163][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:01:39,489][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:01:39,813][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:01:40,141][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:01:40,472][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:01:40,799][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:01:41,127][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:01:41,455][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:01:42,245][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:01:43,019][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:01:43,020][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:01:43,022][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:01:44,000][__main__][INFO] - Iteration 467 took 23s (40.06% Gen, 55.85% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 9s. Estimated total time: 19h 58m 4s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 56s, 500 more iterations: 3h 19m 40s.
[2025-11-13 11:01:44,002][__main__][INFO] - Starting iteration 467.
[2025-11-13 11:01:44,005][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:01:44,005][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:01:54,117][__main__][INFO] - Number of regex retries in iteration 467: 0
[2025-11-13 11:01:54,118][__main__][INFO] - agents played in iteration 467 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:01:54,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:54,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:54,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:54,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:54,720][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:01:54,721][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:01:55,504][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:01:55,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:01:56,131][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:01:56,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:01:56,787][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:01:57,118][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:01:57,437][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:01:57,763][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:01:58,089][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:01:58,416][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:01:58,742][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:01:59,068][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:01:59,394][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:01:59,723][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:00,045][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:00,370][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:00,695][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:01,020][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:01,345][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:02:01,670][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:02:01,996][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:02:02,325][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:02:02,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:02:02,972][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:02:03,298][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:02:03,624][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:02:03,951][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:02:04,276][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:02:04,602][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:02:04,930][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:02:05,254][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:02:05,581][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:02:05,907][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:02:06,686][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:02:07,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:02:07,427][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:02:07,429][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:02:08,410][__main__][INFO] - Iteration 468 took 24s (41.43% Gen, 54.54% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 21m 59s. Estimated total time: 20h 20m 18s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 40s, 500 more iterations: 3h 23m 23s.
[2025-11-13 11:02:08,412][__main__][INFO] - Starting iteration 468.
[2025-11-13 11:02:08,415][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:02:08,415][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:02:18,306][__main__][INFO] - Number of regex retries in iteration 468: 0
[2025-11-13 11:02:18,307][__main__][INFO] - agents played in iteration 468 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:02:18,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:18,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:18,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:18,918][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:18,919][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:02:18,919][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:02:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:02:20,368][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:02:20,696][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:02:21,021][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:02:21,348][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:02:21,673][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:02:21,997][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:02:22,323][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:02:22,650][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:02:22,978][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:02:23,311][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:02:23,631][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:02:23,957][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:02:24,283][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:24,616][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:24,935][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:25,262][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:25,588][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:25,919][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:02:26,241][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:02:26,566][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:02:26,892][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:02:27,226][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:02:27,549][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:02:27,874][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:02:28,199][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:02:28,529][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:02:28,850][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:02:29,176][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:02:29,504][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:02:29,834][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:02:30,155][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:02:30,486][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:02:31,256][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:02:31,989][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:02:31,991][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:02:31,992][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:02:32,957][__main__][INFO] - Iteration 469 took 24s (40.30% Gen, 55.76% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 28m 26s. Estimated total time: 20h 27m 10s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 54s, 500 more iterations: 3h 24m 31s.
[2025-11-13 11:02:32,959][__main__][INFO] - Starting iteration 469.
[2025-11-13 11:02:32,963][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:02:32,963][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:02:42,325][__main__][INFO] - Number of regex retries in iteration 469: 0
[2025-11-13 11:02:42,326][__main__][INFO] - agents played in iteration 469 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:02:42,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:43,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:43,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:43,315][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:43,315][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:02:43,315][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:02:44,098][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:02:44,410][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:02:44,738][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:02:45,065][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:02:45,392][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:02:45,720][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:02:46,046][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:02:46,371][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:02:46,697][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:02:47,025][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:02:47,350][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:02:47,676][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:02:48,003][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:02:48,331][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:48,660][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:48,985][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:49,311][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:49,637][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:49,965][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:02:50,292][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:02:50,619][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:02:50,946][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:02:51,274][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:02:51,600][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:02:51,924][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:02:52,248][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:02:52,573][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:02:52,900][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:02:53,226][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:02:53,555][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:02:53,880][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:02:54,209][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:02:54,537][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:02:55,295][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:02:56,063][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:02:56,064][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:02:56,066][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:02:57,057][__main__][INFO] - Iteration 470 took 24s (38.86% Gen, 57.02% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 5m 39s. Estimated total time: 20h 4m 47s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 9s, 500 more iterations: 3h 20m 47s.
[2025-11-13 11:02:57,060][__main__][INFO] - Starting iteration 470.
[2025-11-13 11:02:57,063][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:02:57,064][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:03:07,191][__main__][INFO] - Number of regex retries in iteration 470: 0
[2025-11-13 11:03:07,192][__main__][INFO] - agents played in iteration 470 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:03:07,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:07,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:07,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:07,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:07,798][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:03:07,799][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:03:08,594][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:03:08,892][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:03:09,226][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:03:09,545][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:03:09,871][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:03:10,195][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:03:10,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:03:10,847][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:03:11,173][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:03:11,498][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:03:11,832][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:03:12,150][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:03:12,480][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:03:12,807][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:03:13,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:03:13,463][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:03:13,790][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:03:14,117][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:03:14,444][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:03:14,767][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:03:15,094][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:03:15,422][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:03:15,749][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:03:16,077][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:03:16,405][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:03:16,730][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:03:17,058][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:03:17,387][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:03:17,712][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:03:18,038][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:03:18,366][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:03:18,691][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:03:19,019][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:03:19,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:03:20,550][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:03:20,551][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:03:20,553][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:03:22,448][__main__][INFO] - Iteration 471 took 25s (39.89% Gen, 52.63% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 9m 44s. Estimated total time: 21h 9m 18s. Time estimates for 10 more iterations: 4m 13s, 100 more iterations: 42m 18s, 500 more iterations: 3h 31m 33s.
[2025-11-13 11:03:22,450][__main__][INFO] - Starting iteration 471.
[2025-11-13 11:03:22,452][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:03:22,453][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:03:32,667][__main__][INFO] - Number of regex retries in iteration 471: 0
[2025-11-13 11:03:32,668][__main__][INFO] - agents played in iteration 471 are Alice, Bob_buffer, Bob, Alice_buffer
[2025-11-13 11:03:33,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:33,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:33,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:33,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:33,272][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:03:33,272][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:03:34,043][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:03:34,340][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:03:34,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:03:34,993][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:03:35,320][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:03:35,646][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:03:35,971][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:03:36,298][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:03:36,624][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:03:36,951][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:03:37,276][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:03:37,603][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:03:37,927][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:03:38,254][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:03:38,584][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:03:38,914][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:03:39,241][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:03:39,572][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:03:39,899][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:03:40,228][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:03:40,554][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:03:40,880][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:03:41,210][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:03:41,533][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:03:41,861][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:03:42,187][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:03:42,514][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:03:42,839][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:03:43,170][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:03:43,498][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:03:43,822][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:03:44,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:03:44,481][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:03:45,269][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:03:45,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:03:45,997][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:03:45,999][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:06:50,941][mllm.models.large_language_model_local][INFO] - Loaded 47 past agent adapters from checkpoints directory.
[2025-11-13 11:07:09,915][mllm.models.large_language_model_local][INFO] - Initializing adapter 'agent_adapter': using existing weights from output folder '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/Qwen/Qwen2.5-7B-Instruct/adapters/agent_adapter'.
[2025-11-13 11:07:11,322][mllm.models.adapter_training_wrapper][INFO] - Adapter 'agent_adapter': loaded initial weights from '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/Qwen/Qwen2.5-7B-Instruct/adapters/agent_adapter'.
[2025-11-13 11:07:11,329][mllm.models.large_language_model_local][INFO] - Initializing adapter 'critic_adapter': using existing weights from output folder '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/Qwen/Qwen2.5-7B-Instruct/adapters/critic_adapter'.
[2025-11-13 11:07:12,163][mllm.models.adapter_training_wrapper][WARNING] - Adapter 'critic_adapter': failed to load from '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/Qwen/Qwen2.5-7B-Instruct/adapters/critic_adapter': Error while deserializing header: HeaderTooSmall
[2025-11-13 11:07:12,164][mllm.models.adapter_training_wrapper][INFO] - Adapter 'critic_adapter': initialized with fresh weights (no initial weights found).
[2025-11-13 11:09:20,829][mllm.training.trainer_common][INFO] - Loading trainer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:09:20,832][mllm.training.trainer_common][INFO] - Loading policy optimizer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:09:21,829][mllm.training.trainer_common][INFO] - Loading critic optimizer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:09:21,832][__main__][INFO] - Starting iteration 471.
[2025-11-13 11:09:21,835][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:09:21,835][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:09:50,630][__main__][INFO] - Number of regex retries in iteration 471: 0
[2025-11-13 11:09:50,631][__main__][INFO] - agents played in iteration 471 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:09:51,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:51,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:51,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:51,219][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:51,220][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:09:51,220][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:09:51,832][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:09:52,447][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:09:52,772][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:09:53,102][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:09:53,430][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:09:53,755][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:09:54,085][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:09:54,408][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:09:54,734][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:09:55,059][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:09:55,382][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:09:55,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:09:56,027][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:09:56,350][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:09:56,673][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:09:56,995][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:09:57,318][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:09:57,639][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:09:57,962][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:09:58,284][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:09:58,607][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:09:58,930][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:09:59,253][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:09:59,578][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:09:59,900][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:10:00,222][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:10:00,545][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:10:00,869][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:10:01,193][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:10:01,519][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:10:01,844][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:02,167][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:02,493][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:03,183][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.78%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:10:03,966][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:10:03,968][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:10:03,971][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:10:05,287][__main__][INFO] - Iteration 472 took 43s (66.27% Gen, 30.70% Train). Generation: 28s, Training: 13s. Estimated remaining time: 36h 9m 23s. Estimated total time: 36h 12m 39s. Time estimates for 10 more iterations: 7m 14s, 100 more iterations: 1h 12m 25s, 500 more iterations: 6h 2m 6s.
[2025-11-13 11:10:05,290][__main__][INFO] - Starting iteration 472.
[2025-11-13 11:10:05,294][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:10:05,294][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:10:19,794][__main__][INFO] - Number of regex retries in iteration 472: 0
[2025-11-13 11:10:19,795][__main__][INFO] - agents played in iteration 472 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:10:20,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,325][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,365][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:10:20,365][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:10:21,078][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:10:21,373][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:10:21,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:10:22,026][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:10:22,352][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:10:22,674][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:10:22,997][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:10:23,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:10:23,644][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:10:23,966][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:10:24,289][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:10:24,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:10:24,935][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:10:25,259][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:10:25,582][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:10:25,905][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:10:26,229][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:10:26,552][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:10:26,878][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:10:27,201][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:10:27,524][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:10:27,846][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:10:28,170][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:10:28,493][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:10:28,816][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:10:29,140][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:10:29,465][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:10:29,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:10:30,116][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:10:30,441][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:10:30,765][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:31,088][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:31,411][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:32,119][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:10:32,859][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:10:32,861][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:10:32,863][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:10:33,793][__main__][INFO] - Iteration 473 took 28s (50.88% Gen, 45.85% Train). Generation: 14s, Training: 13s. Estimated remaining time: 23h 41m 15s. Estimated total time: 23h 45m 0s. Time estimates for 10 more iterations: 4m 45s, 100 more iterations: 47m 30s, 500 more iterations: 3h 57m 30s.
[2025-11-13 11:10:33,795][__main__][INFO] - Starting iteration 473.
[2025-11-13 11:10:33,800][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:10:33,801][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:10:47,705][__main__][INFO] - Number of regex retries in iteration 473: 0
[2025-11-13 11:10:47,706][__main__][INFO] - agents played in iteration 473 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:10:48,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:48,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:48,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:48,282][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:48,282][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:10:48,283][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:10:48,938][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:10:49,233][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:10:49,557][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:10:49,881][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:10:50,204][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:10:50,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:10:50,852][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:10:51,179][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:10:51,502][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:10:51,826][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:10:52,152][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:10:52,475][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:10:52,798][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:10:53,122][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:10:53,448][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:10:53,773][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:10:54,096][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:10:54,420][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:10:54,743][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:10:55,067][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:10:55,391][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:10:55,714][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:10:56,038][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:10:56,362][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:10:56,685][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:10:57,009][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:10:57,333][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:10:57,657][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:10:57,982][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:10:58,306][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:10:58,633][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:58,960][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:59,284][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:59,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:11:00,674][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:11:00,676][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:11:00,678][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:11:01,600][__main__][INFO] - Iteration 474 took 27s (50.02% Gen, 46.66% Train). Generation: 13s, Training: 12s. Estimated remaining time: 23h 5m 49s. Estimated total time: 23h 10m 1s. Time estimates for 10 more iterations: 4m 38s, 100 more iterations: 46m 20s, 500 more iterations: 3h 51m 40s.
[2025-11-13 11:11:01,601][__main__][INFO] - Starting iteration 474.
[2025-11-13 11:11:01,604][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:11:01,605][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:11:12,580][__main__][INFO] - Number of regex retries in iteration 474: 0
[2025-11-13 11:11:12,581][__main__][INFO] - agents played in iteration 474 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:11:13,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:13,056][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:13,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:13,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:13,136][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:11:13,137][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:11:13,807][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:11:14,105][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:11:14,429][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:11:14,754][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:11:15,077][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:11:15,400][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:11:15,723][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:11:16,049][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:11:16,372][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:11:16,695][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:11:17,017][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:11:17,340][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:11:17,662][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:11:17,985][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:11:18,308][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:11:18,632][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:11:18,955][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:11:19,278][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:11:19,601][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:11:19,924][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:11:20,247][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:11:20,571][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:11:20,895][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:11:21,219][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:11:21,542][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:11:21,866][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:11:22,190][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:11:22,511][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:11:22,834][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:11:23,157][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:11:23,480][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:11:23,808][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:11:24,132][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:11:24,801][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:11:25,525][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:11:25,527][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:11:25,529][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:11:26,522][__main__][INFO] - Iteration 475 took 24s (44.05% Gen, 51.96% Train). Generation: 10s, Training: 12s. Estimated remaining time: 20h 41m 19s. Estimated total time: 20h 45m 57s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 31s, 500 more iterations: 3h 27m 39s.
[2025-11-13 11:11:26,524][__main__][INFO] - Starting iteration 475.
[2025-11-13 11:11:26,527][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:11:26,528][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:11:36,620][__main__][INFO] - Number of regex retries in iteration 475: 0
[2025-11-13 11:11:36,621][__main__][INFO] - agents played in iteration 475 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:11:37,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:11:37,177][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:11:37,874][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:11:38,171][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:11:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:11:38,823][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:11:39,147][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:11:39,471][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:11:39,795][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:11:40,122][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:11:40,445][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:11:40,771][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:11:41,094][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:11:41,419][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:11:41,744][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:11:42,067][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:11:42,393][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:11:42,718][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:11:43,042][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:11:43,367][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:11:43,696][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:11:44,024][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:11:44,350][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:11:44,673][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:11:44,997][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:11:45,320][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:11:45,643][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:11:45,967][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:11:46,290][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:11:46,614][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:11:46,937][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:11:47,260][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:11:47,584][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:11:47,908][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:11:48,232][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:11:48,892][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:11:49,630][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:11:49,631][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:11:49,633][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:11:50,518][__main__][INFO] - Iteration 476 took 23s (42.07% Gen, 54.23% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 54m 33s. Estimated total time: 19h 59m 34s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 59s, 500 more iterations: 3h 19m 55s.
[2025-11-13 11:11:50,520][__main__][INFO] - Starting iteration 476.
[2025-11-13 11:11:50,523][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:11:50,524][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:01,983][__main__][INFO] - Number of regex retries in iteration 476: 0
[2025-11-13 11:12:01,984][__main__][INFO] - agents played in iteration 476 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:12:02,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,533][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:02,533][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:03,218][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:03,516][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:03,841][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:04,165][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:04,489][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:04,812][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:05,137][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:05,460][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:05,783][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:06,109][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:06,436][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:06,768][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:07,095][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:07,427][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:07,750][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:08,075][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:08,406][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:08,730][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:09,056][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:09,379][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:09,702][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:10,025][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:12:10,352][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:12:10,684][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:12:11,006][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:12:11,330][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:12:11,654][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:12:11,977][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:12:12,300][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:12:12,623][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:12:12,947][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:12:13,274][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:12:13,602][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:12:14,280][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:12:15,022][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:12:15,024][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:12:15,026][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:12:16,038][__main__][INFO] - Iteration 477 took 25s (44.91% Gen, 51.11% Train). Generation: 11s, Training: 13s. Estimated remaining time: 21h 10m 19s. Estimated total time: 21h 15m 46s. Time estimates for 10 more iterations: 4m 15s, 100 more iterations: 42m 31s, 500 more iterations: 3h 32m 37s.
[2025-11-13 11:12:16,040][__main__][INFO] - Starting iteration 477.
[2025-11-13 11:12:16,043][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:12:16,043][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:25,314][__main__][INFO] - Number of regex retries in iteration 477: 0
[2025-11-13 11:12:25,315][__main__][INFO] - agents played in iteration 477 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:12:25,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:25,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:25,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:25,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:25,873][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:25,874][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:26,583][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:26,882][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:27,213][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:27,539][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:27,866][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:28,193][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:28,518][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:28,850][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:29,177][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:29,509][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:29,840][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:30,168][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:30,498][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:30,821][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:31,147][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:31,471][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:31,795][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:32,126][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:32,450][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:32,775][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:33,098][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:33,425][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:12:33,750][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:12:34,073][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:12:34,397][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:12:34,720][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:12:35,045][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:12:35,372][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:12:35,704][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:12:36,028][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:12:36,361][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:12:36,685][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:12:37,008][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:12:37,686][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:12:38,441][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:12:38,443][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:12:38,444][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:12:39,542][__main__][INFO] - Iteration 478 took 23s (39.45% Gen, 55.87% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 29m 9s. Estimated total time: 19h 35m 0s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 50s.
[2025-11-13 11:12:39,544][__main__][INFO] - Starting iteration 478.
[2025-11-13 11:12:39,548][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:12:39,549][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:49,132][__main__][INFO] - Number of regex retries in iteration 478: 0
[2025-11-13 11:12:49,133][__main__][INFO] - agents played in iteration 478 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:12:49,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:49,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:49,646][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:49,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:49,686][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:49,687][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:50,377][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:50,674][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:51,000][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:51,330][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:51,660][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:51,984][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:52,312][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:52,636][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:53,295][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:53,622][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:53,945][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:54,270][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:54,594][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:54,918][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:55,243][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:55,573][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:55,900][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:56,226][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:56,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:56,878][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:57,201][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:12:57,525][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:12:57,850][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:12:58,173][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:12:58,496][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:12:58,821][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:12:59,147][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:12:59,474][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:12:59,800][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:00,127][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:00,453][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:00,777][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:01,457][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:02,208][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:02,209][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:02,211][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:03,113][__main__][INFO] - Iteration 479 took 23s (40.66% Gen, 55.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 32m 7s. Estimated total time: 19h 38m 21s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 23s.
[2025-11-13 11:13:03,115][__main__][INFO] - Starting iteration 479.
[2025-11-13 11:13:03,118][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:13:03,119][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:13:12,973][__main__][INFO] - Number of regex retries in iteration 479: 0
[2025-11-13 11:13:12,974][__main__][INFO] - agents played in iteration 479 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:13:13,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:13,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:13,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:13,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:13,533][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:13:13,534][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:13:14,210][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:13:14,507][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:13:14,833][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:13:15,158][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:13:15,486][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:13:15,810][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:13:16,138][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:13:16,469][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:13:16,803][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:13:17,127][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:13:17,457][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:13:17,783][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:13:18,109][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:13:18,436][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:13:18,763][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:13:19,089][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:13:19,416][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:13:19,741][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:13:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:13:20,395][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:13:20,718][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:13:21,049][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:13:21,372][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:13:21,700][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:13:22,034][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:13:22,362][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:13:22,689][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:13:23,014][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:13:23,341][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:13:23,664][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:23,994][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:24,316][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:24,642][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:25,281][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:26,006][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:26,007][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:26,009][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:26,931][__main__][INFO] - Iteration 480 took 23s (41.38% Gen, 54.74% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 44m 2s. Estimated total time: 19h 50m 40s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 26s.
[2025-11-13 11:13:26,933][__main__][INFO] - Starting iteration 480.
[2025-11-13 11:13:26,936][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:13:26,937][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:13:36,533][__main__][INFO] - Number of regex retries in iteration 480: 0
[2025-11-13 11:13:36,533][__main__][INFO] - agents played in iteration 480 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:13:36,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:37,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:37,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:37,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:37,087][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:13:37,087][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:13:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:13:38,074][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:13:38,399][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:13:38,725][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:13:39,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:13:39,378][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:13:39,703][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:13:40,028][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:13:40,353][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:13:40,678][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:13:41,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:13:41,332][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:13:41,658][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:13:41,988][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:13:42,316][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:13:42,642][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:13:42,966][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:13:43,290][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:13:43,615][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:13:43,939][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:13:44,264][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:13:44,588][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:13:44,912][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:13:45,237][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:13:45,561][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:13:45,885][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:13:46,209][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:13:46,535][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:13:46,859][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:13:47,186][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:47,510][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:47,833][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:48,161][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:48,809][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:49,541][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:49,543][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:49,544][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:51,754][__main__][INFO] - Iteration 481 took 24s (38.66% Gen, 52.43% Train). Generation: 9s, Training: 13s. Estimated remaining time: 20h 33m 53s. Estimated total time: 20h 40m 56s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 21s, 500 more iterations: 3h 26m 49s.
[2025-11-13 11:13:51,756][__main__][INFO] - Starting iteration 481.
[2025-11-13 11:13:51,759][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:13:51,760][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:01,654][__main__][INFO] - Number of regex retries in iteration 481: 0
[2025-11-13 11:14:01,655][__main__][INFO] - agents played in iteration 481 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:14:02,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:02,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:02,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:02,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:02,215][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:02,216][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:02,910][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:03,208][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:03,535][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:03,861][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:04,186][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:04,510][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:04,841][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:05,166][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:05,492][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:05,818][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:06,145][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:06,470][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:06,797][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:07,127][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:07,455][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:07,782][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:08,107][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:08,432][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:08,756][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:14:09,079][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:14:09,406][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:14:09,731][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:14:10,055][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:14:10,378][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:14:10,703][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:14:11,028][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:14:11,351][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:14:11,675][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:14:12,000][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:14:12,323][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:14:12,648][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:14:12,972][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:14:13,296][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:14:13,936][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:14:14,663][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:14:14,664][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:14:14,666][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:14:15,693][__main__][INFO] - Iteration 482 took 23s (41.34% Gen, 54.36% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 49m 16s. Estimated total time: 19h 56m 42s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 27s.
[2025-11-13 11:14:15,695][__main__][INFO] - Starting iteration 482.
[2025-11-13 11:14:15,698][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:14:15,699][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:24,825][__main__][INFO] - Number of regex retries in iteration 482: 0
[2025-11-13 11:14:24,825][__main__][INFO] - agents played in iteration 482 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:14:25,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:25,300][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:25,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:25,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:25,381][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:25,381][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:26,082][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:26,378][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:26,709][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:27,041][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:27,375][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:27,707][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:28,036][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:28,360][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:28,687][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:29,013][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:29,346][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:29,672][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:30,001][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:30,329][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:30,657][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:30,983][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:31,309][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:31,636][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:31,964][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:14:32,288][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:14:32,618][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:14:32,942][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:14:33,269][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:14:33,595][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:14:33,919][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:14:34,251][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:14:34,579][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:14:34,903][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:14:35,234][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:14:35,558][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:14:35,882][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:14:36,209][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:14:36,534][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:14:37,174][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:14:37,917][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:14:37,919][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:14:37,920][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:14:38,972][__main__][INFO] - Iteration 483 took 23s (39.21% Gen, 56.26% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 15m 55s. Estimated total time: 19h 23m 45s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 57s.
[2025-11-13 11:14:38,974][__main__][INFO] - Starting iteration 483.
[2025-11-13 11:14:38,978][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:14:38,978][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:48,958][__main__][INFO] - Number of regex retries in iteration 483: 0
[2025-11-13 11:14:48,959][__main__][INFO] - agents played in iteration 483 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:14:49,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:49,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:49,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:49,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:49,513][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:49,514][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:50,233][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:50,530][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:50,858][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:51,190][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:51,524][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:51,851][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:52,181][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:52,512][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:52,842][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:53,171][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:53,500][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:53,828][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:54,155][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:54,482][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:54,813][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:55,137][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:55,465][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:55,792][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:56,119][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:14:56,446][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:14:56,774][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:14:57,105][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:14:57,432][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:14:57,758][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:14:58,085][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:14:58,412][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:14:58,735][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:14:59,059][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:14:59,383][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:14:59,707][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:00,033][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:00,357][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:00,681][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:01,349][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:02,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:02,106][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:02,108][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:03,016][__main__][INFO] - Iteration 484 took 24s (41.51% Gen, 54.70% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 53m 44s. Estimated total time: 20h 1m 58s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 19s.
[2025-11-13 11:15:03,018][__main__][INFO] - Starting iteration 484.
[2025-11-13 11:15:03,021][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:03,021][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:15:09,043][mllm.models.large_language_model_local][WARNING] - Response \<A\> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 11:15:12,740][__main__][INFO] - Number of regex retries in iteration 484: 1
[2025-11-13 11:15:12,740][__main__][INFO] - agents played in iteration 484 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:15:13,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:13,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:13,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:13,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:13,604][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:15:13,604][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:15:14,288][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:15:14,583][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:15:14,906][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:15:15,232][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:15:15,556][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:15:15,883][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:15:16,210][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:15:16,533][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:15:16,862][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:15:17,188][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:15:17,511][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:15:17,840][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:15:18,164][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:15:18,486][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:15:18,811][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:15:19,133][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:15:19,454][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:15:19,780][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:15:20,102][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:15:20,425][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:15:20,748][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:15:21,075][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:15:21,395][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:15:21,716][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:15:22,038][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:15:22,360][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:15:22,686][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:15:23,011][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:15:23,335][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:15:23,657][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:23,978][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:24,300][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:24,623][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:25,293][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:26,030][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:26,031][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:26,033][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:26,900][__main__][INFO] - Iteration 485 took 23s (40.70% Gen, 55.66% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 45m 21s. Estimated total time: 19h 53m 59s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 47s, 500 more iterations: 3h 18m 59s.
[2025-11-13 11:15:26,902][__main__][INFO] - Starting iteration 485.
[2025-11-13 11:15:26,906][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:26,907][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:15:36,348][__main__][INFO] - Number of regex retries in iteration 485: 0
[2025-11-13 11:15:36,349][__main__][INFO] - agents played in iteration 485 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:15:36,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:36,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:36,890][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:36,924][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:36,925][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:15:36,925][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:15:37,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:15:37,915][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:15:38,243][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:15:38,568][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:15:38,896][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:15:39,218][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:15:39,546][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:15:39,874][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:15:40,199][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:15:40,525][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:15:40,851][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:15:41,175][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:15:41,501][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:15:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:15:42,155][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:15:42,480][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:15:42,803][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:15:43,125][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:15:43,446][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:15:43,767][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:15:44,089][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:15:44,414][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:15:44,736][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:15:45,058][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:15:45,380][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:15:45,702][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:15:46,023][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:15:46,344][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:15:46,666][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:15:46,987][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:47,308][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:47,629][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:47,951][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:48,629][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:49,365][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:49,367][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:49,369][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:50,448][__main__][INFO] - Iteration 486 took 23s (40.11% Gen, 55.30% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 28m 10s. Estimated total time: 19h 37m 11s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 11s.
[2025-11-13 11:15:50,450][__main__][INFO] - Starting iteration 486.
[2025-11-13 11:15:50,454][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:50,455][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:00,529][__main__][INFO] - Number of regex retries in iteration 486: 0
[2025-11-13 11:16:00,530][__main__][INFO] - agents played in iteration 486 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:16:00,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:01,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:01,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:01,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:01,068][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:01,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:01,755][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:02,050][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:02,373][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:02,695][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:03,021][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:03,348][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:03,671][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:03,995][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:04,319][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:04,641][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:04,965][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:05,288][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:05,615][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:05,940][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:06,262][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:06,585][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:06,906][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:07,230][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:07,549][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:07,871][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:08,193][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:08,514][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:08,835][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:09,158][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:16:09,480][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:16:09,801][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:16:10,122][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:16:10,445][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:16:10,767][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:16:11,089][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:16:11,411][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:16:11,734][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:16:12,057][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:16:12,785][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:16:13,521][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:16:13,523][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:16:13,524][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:16:14,470][__main__][INFO] - Iteration 487 took 24s (41.95% Gen, 54.11% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 51m 23s. Estimated total time: 20h 0m 49s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 1s, 500 more iterations: 3h 20m 8s.
[2025-11-13 11:16:14,472][__main__][INFO] - Starting iteration 487.
[2025-11-13 11:16:14,475][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:16:14,475][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:24,302][__main__][INFO] - Number of regex retries in iteration 487: 0
[2025-11-13 11:16:24,303][__main__][INFO] - agents played in iteration 487 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:16:24,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:24,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:24,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:24,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:24,839][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:24,840][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:25,516][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:25,811][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:26,134][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:26,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:26,781][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:27,106][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:27,429][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:28,076][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:28,399][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:28,721][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:29,046][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:29,368][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:29,694][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:30,016][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:30,338][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:30,661][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:30,986][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:31,311][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:31,634][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:31,955][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:32,279][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:32,602][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:32,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:16:33,248][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:16:33,573][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:16:33,895][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:16:34,218][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:16:34,540][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:16:34,862][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:16:35,185][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:16:35,510][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:16:35,835][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:16:36,506][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:16:37,226][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:16:37,228][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:16:37,229][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:16:38,203][__main__][INFO] - Iteration 488 took 23s (41.41% Gen, 54.48% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 36m 37s. Estimated total time: 19h 46m 26s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 32s, 500 more iterations: 3h 17m 44s.
[2025-11-13 11:16:38,205][__main__][INFO] - Starting iteration 488.
[2025-11-13 11:16:38,208][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:16:38,208][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:47,812][__main__][INFO] - Number of regex retries in iteration 488: 0
[2025-11-13 11:16:47,813][__main__][INFO] - agents played in iteration 488 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:16:48,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:48,284][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:48,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:48,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:48,350][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:48,350][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:49,026][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:49,321][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:49,645][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:49,967][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:50,292][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:50,617][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:50,945][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:51,272][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:51,595][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:51,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:52,249][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:52,571][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:52,894][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:53,217][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:53,539][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:53,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:54,187][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:54,510][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:54,831][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:55,152][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:55,474][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:55,796][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:56,118][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:56,439][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:16:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:16:57,086][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:16:57,411][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:16:57,732][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:16:58,053][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:16:58,380][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:16:58,702][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:16:59,024][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:16:59,347][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:00,017][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:17:00,734][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:00,735][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:00,737][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:01,721][__main__][INFO] - Iteration 489 took 23s (40.84% Gen, 54.97% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 25m 28s. Estimated total time: 19h 35m 41s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 56s.
[2025-11-13 11:17:01,723][__main__][INFO] - Starting iteration 489.
[2025-11-13 11:17:01,726][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:17:01,726][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:11,308][__main__][INFO] - Number of regex retries in iteration 489: 0
[2025-11-13 11:17:11,309][__main__][INFO] - agents played in iteration 489 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:17:11,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:11,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:11,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:11,865][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:11,866][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:17:11,866][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:17:12,573][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:17:12,867][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:17:13,194][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:17:13,518][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:17:13,840][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:17:14,162][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:17:14,484][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:17:14,807][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:17:15,132][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:17:15,454][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:17:15,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:17:16,103][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:17:16,428][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:17:16,750][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:17:17,075][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:17:17,399][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:17:17,722][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:17:18,043][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:17:18,363][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:17:18,684][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:17:19,006][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:17:19,330][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:17:19,652][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:17:19,973][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:17:20,295][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:17:20,616][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:17:20,938][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:17:21,259][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:17:21,580][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:17:21,904][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:17:22,225][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:17:22,546][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:17:22,868][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:23,559][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:17:24,281][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:24,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:24,284][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:25,291][__main__][INFO] - Iteration 490 took 23s (40.66% Gen, 55.06% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 27m 41s. Estimated total time: 19h 38m 17s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 22s.
[2025-11-13 11:17:25,293][__main__][INFO] - Starting iteration 490.
[2025-11-13 11:17:25,296][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:17:25,297][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:34,094][__main__][INFO] - Number of regex retries in iteration 490: 0
[2025-11-13 11:17:34,094][__main__][INFO] - agents played in iteration 490 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:17:34,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:34,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:34,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:34,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:34,641][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:17:34,642][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:17:35,344][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:17:35,811][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:17:36,134][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:17:36,456][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:17:36,781][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:17:37,109][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:17:37,433][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:17:37,759][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:17:38,086][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:17:38,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:17:38,735][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:17:39,062][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:17:39,388][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:17:39,718][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:17:40,044][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:17:40,367][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:17:40,692][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:17:41,016][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:17:41,343][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:17:41,666][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:17:41,987][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:17:42,311][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:17:42,632][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:17:42,956][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:17:43,280][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:17:43,601][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:17:43,927][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:17:44,251][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:17:44,573][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:17:44,896][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:17:45,218][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:17:45,538][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:17:45,859][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:46,540][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:17:47,262][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:47,264][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:47,266][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:49,124][__main__][INFO] - Iteration 491 took 23s (36.92% Gen, 55.28% Train). Generation: 8s, Training: 13s. Estimated remaining time: 19h 40m 26s. Estimated total time: 19h 51m 26s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 42s, 500 more iterations: 3h 18m 34s.
[2025-11-13 11:17:49,126][__main__][INFO] - Starting iteration 491.
[2025-11-13 11:17:49,129][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:17:49,130][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:58,810][__main__][INFO] - Number of regex retries in iteration 491: 0
[2025-11-13 11:17:58,811][__main__][INFO] - agents played in iteration 491 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:17:59,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:59,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:59,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:59,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:59,345][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:17:59,345][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:00,049][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:00,343][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:00,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:00,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:01,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:01,640][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:01,961][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:02,285][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:02,608][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:02,930][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:03,253][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:03,576][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:03,898][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:04,220][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:04,543][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:04,871][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:05,199][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:05,522][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:05,846][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:06,170][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:06,493][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:06,816][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:07,140][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:07,461][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:07,782][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:08,103][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:08,426][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:08,749][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:09,073][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:09,397][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:18:09,723][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:18:10,050][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:18:10,375][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:18:11,071][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:18:11,799][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:18:11,801][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:18:11,802][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:18:12,718][__main__][INFO] - Iteration 492 took 23s (41.04% Gen, 55.07% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 28m 6s. Estimated total time: 19h 39m 30s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 35s.
[2025-11-13 11:18:12,720][__main__][INFO] - Starting iteration 492.
[2025-11-13 11:18:12,722][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:18:12,723][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:18:22,048][__main__][INFO] - Number of regex retries in iteration 492: 0
[2025-11-13 11:18:22,049][__main__][INFO] - agents played in iteration 492 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:18:22,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:22,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:22,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:22,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:22,582][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:18:22,582][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:23,280][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:23,573][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:23,895][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:24,220][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:24,548][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:24,870][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:25,197][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:25,523][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:25,846][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:26,170][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:26,492][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:26,815][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:27,139][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:27,462][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:27,784][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:28,106][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:28,431][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:28,754][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:29,075][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:29,397][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:29,718][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:30,042][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:30,364][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:30,687][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:31,009][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:31,331][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:31,653][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:31,974][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:32,296][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:32,617][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:18:32,939][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:18:33,261][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:18:33,583][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:18:34,257][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:18:34,974][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:18:34,977][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:18:34,979][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:18:35,950][__main__][INFO] - Iteration 493 took 23s (40.15% Gen, 55.66% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 9m 39s. Estimated total time: 19h 21m 26s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 34s.
[2025-11-13 11:18:35,952][__main__][INFO] - Starting iteration 493.
[2025-11-13 11:18:35,955][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:18:35,955][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:18:44,853][__main__][INFO] - Number of regex retries in iteration 493: 0
[2025-11-13 11:18:44,854][__main__][INFO] - agents played in iteration 493 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:18:45,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:45,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:45,357][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:45,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:45,392][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:18:45,392][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:46,098][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:46,393][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:46,716][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:47,039][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:47,362][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:47,684][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:48,008][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:48,329][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:48,651][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:48,977][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:49,299][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:49,622][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:49,947][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:50,269][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:50,592][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:50,919][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:51,243][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:51,567][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:51,888][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:52,212][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:52,534][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:52,855][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:53,176][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:53,498][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:53,819][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:54,139][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:54,461][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:54,783][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:55,104][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:55,426][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:18:55,747][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:18:56,068][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:18:56,390][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:18:57,073][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:18:57,792][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:18:57,793][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:18:57,795][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:18:58,789][__main__][INFO] - Iteration 494 took 22s (38.97% Gen, 56.67% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 49m 35s. Estimated total time: 19h 1m 45s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 3s, 500 more iterations: 3h 10m 17s.
[2025-11-13 11:18:58,791][__main__][INFO] - Starting iteration 494.
[2025-11-13 11:18:58,794][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:18:58,795][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:07,725][__main__][INFO] - Number of regex retries in iteration 494: 0
[2025-11-13 11:19:07,725][__main__][INFO] - agents played in iteration 494 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:19:08,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:08,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:08,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:08,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:08,282][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:19:08,283][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:19:09,999][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:19:09,294][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:19:09,618][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:19:09,940][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:19:10,267][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:19:10,589][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:19:10,914][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:19:11,242][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:19:11,564][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:19:11,886][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:19:12,209][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:19:12,533][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:19:12,856][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:19:13,182][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:19:13,505][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:19:13,831][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:19:14,154][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:19:14,477][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:19:14,804][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:19:15,126][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:19:15,448][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:19:15,776][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:19:16,100][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:19:16,423][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:19:16,744][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:19:17,071][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:19:17,395][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:19:17,716][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:19:18,039][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:19:18,363][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:19:18,685][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:19:19,008][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:19:19,336][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:19:20,044][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:19:20,769][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:19:20,771][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:19:20,773][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:19:21,776][__main__][INFO] - Iteration 495 took 22s (38.86% Gen, 56.77% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 56m 35s. Estimated total time: 19h 9m 7s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 18s, 500 more iterations: 3h 11m 31s.
[2025-11-13 11:19:21,778][__main__][INFO] - Starting iteration 495.
[2025-11-13 11:19:21,781][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:19:21,781][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:31,028][__main__][INFO] - Number of regex retries in iteration 495: 0
[2025-11-13 11:19:31,028][__main__][INFO] - agents played in iteration 495 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:19:31,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:31,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:31,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:31,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:31,573][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:19:31,574][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:19:32,290][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:19:32,585][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:19:32,908][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:19:33,229][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:19:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:19:33,873][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:19:34,195][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:19:34,518][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:19:34,841][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:19:35,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:19:35,485][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:19:35,808][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:19:36,135][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:19:36,460][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:19:36,787][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:19:37,111][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:19:37,432][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:19:37,754][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:19:38,076][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:19:38,397][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:19:38,718][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:19:39,043][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:19:39,363][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:19:39,686][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:19:40,007][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:19:40,326][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:19:40,648][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:19:40,968][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:19:41,292][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:19:41,614][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:19:41,935][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:19:42,256][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:19:42,583][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:19:43,259][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:19:43,992][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:19:43,993][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:19:43,995][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:19:44,962][__main__][INFO] - Iteration 496 took 23s (39.89% Gen, 55.93% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 6m 8s. Estimated total time: 19h 19m 4s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 10s.
[2025-11-13 11:19:44,964][__main__][INFO] - Starting iteration 496.
[2025-11-13 11:19:44,967][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:19:44,968][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:52,688][mllm.models.large_language_model_local][WARNING] - Response %A> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 11:19:54,939][__main__][INFO] - Number of regex retries in iteration 496: 1
[2025-11-13 11:19:54,940][__main__][INFO] - agents played in iteration 496 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:19:55,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:55,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:55,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:55,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:55,536][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:19:55,537][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:19:56,241][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:19:56,537][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:19:56,860][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:19:57,181][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:19:57,502][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:19:57,824][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:19:58,147][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:19:58,470][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:19:58,793][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:19:59,114][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:19:59,442][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:19:59,765][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:00,087][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:00,409][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:00,732][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:01,054][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:01,375][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:01,698][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:02,024][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:02,345][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:02,666][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:02,990][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:03,311][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:03,632][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:03,955][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:04,278][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:04,598][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:04,920][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:05,241][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:05,564][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:05,886][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:06,209][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:06,532][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:07,214][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:20:07,952][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:07,954][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:07,955][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:20:08,933][__main__][INFO] - Iteration 497 took 23s (41.61% Gen, 54.31% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 45m 0s. Estimated total time: 19h 58m 20s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 56s, 500 more iterations: 3h 19m 43s.
[2025-11-13 11:20:08,935][__main__][INFO] - Starting iteration 497.
[2025-11-13 11:20:08,938][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:20:08,939][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:20:18,323][__main__][INFO] - Number of regex retries in iteration 497: 0
[2025-11-13 11:20:18,323][__main__][INFO] - agents played in iteration 497 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:20:18,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:18,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:18,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:18,906][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:18,906][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:20:18,907][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:20:19,614][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:20:19,908][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:20:20,231][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:20:20,555][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:20:20,880][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:20:21,207][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:20:21,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:20:21,854][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:20:22,181][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:20:22,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:20:22,826][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:20:23,149][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:23,472][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:23,796][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:24,118][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:24,441][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:24,764][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:25,086][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:25,407][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:25,729][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:26,051][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:26,372][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:26,694][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:27,015][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:27,337][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:27,659][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:27,986][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:28,307][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:28,629][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:28,951][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:29,275][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:29,597][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:29,919][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:30,606][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:20:31,347][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:31,348][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:31,351][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:20:32,367][__main__][INFO] - Iteration 498 took 23s (40.05% Gen, 55.60% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 17m 46s. Estimated total time: 19h 31m 29s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 2s, 500 more iterations: 3h 15m 14s.
[2025-11-13 11:20:32,369][__main__][INFO] - Starting iteration 498.
[2025-11-13 11:20:32,373][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:20:32,373][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:20:41,571][__main__][INFO] - Number of regex retries in iteration 498: 0
[2025-11-13 11:20:41,571][__main__][INFO] - agents played in iteration 498 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:20:42,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:42,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:42,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:42,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:42,134][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:20:42,135][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:20:42,840][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:20:43,136][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:20:43,460][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:20:43,781][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:20:44,103][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:20:44,427][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:20:44,749][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:20:45,073][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:20:45,397][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:20:45,722][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:20:46,044][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:20:46,365][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:46,688][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:47,010][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:47,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:47,655][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:47,978][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:48,298][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:48,621][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:48,943][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:49,264][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:49,588][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:49,909][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:50,230][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:50,552][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:50,874][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:51,195][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:51,516][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:51,837][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:52,480][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:52,803][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:53,132][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:53,816][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:20:54,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:54,530][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:54,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:20:55,562][__main__][INFO] - Iteration 499 took 23s (39.66% Gen, 55.89% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 5m 22s. Estimated total time: 19h 19m 29s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 11:20:55,564][__main__][INFO] - Starting iteration 499.
[2025-11-13 11:20:55,567][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:20:55,567][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:04,791][__main__][INFO] - Number of regex retries in iteration 499: 0
[2025-11-13 11:21:04,791][__main__][INFO] - agents played in iteration 499 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:21:05,259][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:05,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:05,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:05,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:05,361][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:05,361][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:06,070][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:06,364][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:06,687][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:07,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:07,337][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:07,658][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:07,983][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:21:08,304][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:21:08,626][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:21:08,947][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:21:09,269][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:21:09,591][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:21:09,913][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:21:10,235][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:21:10,558][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:21:10,881][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:21:11,204][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:21:11,527][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:21:11,849][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:21:12,171][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:21:12,494][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:21:12,816][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:21:13,137][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:21:13,458][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:21:13,779][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:21:14,100][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:21:14,421][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:21:14,742][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:21:15,062][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:21:15,384][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:21:15,705][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:21:16,026][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:21:16,348][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:21:17,046][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:21:17,760][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:21:17,761][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:21:17,763][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:21:18,815][__main__][INFO] - Iteration 500 took 23s (39.67% Gen, 55.79% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 7m 57s. Estimated total time: 19h 22m 27s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 44s.
[2025-11-13 11:21:18,817][__main__][INFO] - Starting iteration 500.
[2025-11-13 11:21:18,820][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:21:18,820][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:28,318][__main__][INFO] - Number of regex retries in iteration 500: 0
[2025-11-13 11:21:28,319][__main__][INFO] - agents played in iteration 500 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:21:28,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:28,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:28,869][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:28,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:28,903][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:28,904][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:29,610][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:29,904][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:30,227][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:30,548][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:30,872][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:31,194][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:31,517][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:21:31,839][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:21:32,160][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:21:32,483][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:21:32,804][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:21:33,126][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:21:33,449][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:21:33,774][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:21:34,097][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:21:34,418][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:21:34,740][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:21:35,063][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:21:35,391][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:21:35,716][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:21:36,041][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:21:36,365][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:21:36,687][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:21:37,012][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:21:37,336][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:21:37,657][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:21:37,981][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:21:38,302][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:21:38,623][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:21:38,944][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:21:39,266][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:21:39,590][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:21:39,911][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:21:40,591][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:21:41,302][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:21:41,303][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:21:41,307][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:21:43,203][__main__][INFO] - Iteration 501 took 24s (38.95% Gen, 53.27% Train). Generation: 9s, Training: 12s. Estimated remaining time: 20h 4m 18s. Estimated total time: 20h 19m 12s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 38s, 500 more iterations: 3h 23m 12s.
[2025-11-13 11:21:43,205][__main__][INFO] - Starting iteration 501.
[2025-11-13 11:21:43,209][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:21:43,209][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:52,320][__main__][INFO] - Number of regex retries in iteration 501: 0
[2025-11-13 11:21:52,320][__main__][INFO] - agents played in iteration 501 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:21:52,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:52,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:52,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:52,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:52,909][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:52,910][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:53,623][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:53,917][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:54,240][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:54,563][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:54,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:55,208][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:55,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:21:55,852][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:21:56,174][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:21:56,498][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:21:56,819][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:21:57,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:21:57,467][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:21:57,790][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:21:58,113][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:21:58,435][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:21:58,756][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:21:59,077][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:21:59,399][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:21:59,721][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:00,366][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:00,690][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:01,011][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:01,332][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:01,654][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:01,976][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:02,298][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:02,619][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:02,941][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:03,263][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:03,585][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:03,907][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:04,599][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:22:05,295][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:05,297][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:05,299][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:06,238][__main__][INFO] - Iteration 502 took 23s (39.56% Gen, 56.35% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 56m 14s. Estimated total time: 19h 11m 31s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 55s.
[2025-11-13 11:22:06,240][__main__][INFO] - Starting iteration 502.
[2025-11-13 11:22:06,243][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:06,244][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:22:15,590][__main__][INFO] - Number of regex retries in iteration 502: 0
[2025-11-13 11:22:15,591][__main__][INFO] - agents played in iteration 502 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:22:16,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:16,079][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:16,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:16,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:16,146][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:22:16,147][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:22:16,861][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:22:17,154][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:22:17,478][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:22:17,800][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:22:18,129][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:22:18,452][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:22:18,775][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:22:19,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:22:19,422][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:22:19,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:22:20,071][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:22:20,395][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:22:20,718][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:22:21,042][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:22:21,368][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:22:21,689][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:22:22,010][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:22:22,331][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:22:22,656][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:22:22,979][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:23,303][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:23,625][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:23,948][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:24,271][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:24,595][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:24,918][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:25,245][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:25,566][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:25,894][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:26,223][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:26,551][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:26,875][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:27,202][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:27,890][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:22:28,570][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:28,571][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:28,573][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:29,534][__main__][INFO] - Iteration 503 took 23s (40.13% Gen, 55.74% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 8m 56s. Estimated total time: 19h 24m 37s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 49s, 500 more iterations: 3h 14m 6s.
[2025-11-13 11:22:29,536][__main__][INFO] - Starting iteration 503.
[2025-11-13 11:22:29,539][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:29,540][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:22:38,311][__main__][INFO] - Number of regex retries in iteration 503: 0
[2025-11-13 11:22:38,312][__main__][INFO] - agents played in iteration 503 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:22:38,758][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:39,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:39,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:39,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:39,186][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:22:39,187][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:22:39,891][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:22:40,185][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:22:40,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:22:40,830][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:22:41,153][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:22:41,476][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:22:41,802][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:22:42,124][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:22:42,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:22:42,774][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:22:43,100][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:22:43,423][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:22:43,745][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:22:44,073][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:22:44,400][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:22:44,723][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:22:45,051][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:22:45,375][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:22:45,698][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:22:46,022][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:46,346][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:46,670][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:46,995][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:47,317][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:47,645][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:47,968][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:48,290][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:48,612][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:48,935][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:49,258][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:49,580][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:49,902][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:50,224][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:50,918][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:22:51,591][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:51,592][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:51,594][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:52,491][__main__][INFO] - Iteration 504 took 22s (38.22% Gen, 57.87% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 51m 34s. Estimated total time: 19h 7m 37s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 16s.
[2025-11-13 11:22:52,493][__main__][INFO] - Starting iteration 504.
[2025-11-13 11:22:52,496][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:52,496][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:01,219][__main__][INFO] - Number of regex retries in iteration 504: 0
[2025-11-13 11:23:01,220][__main__][INFO] - agents played in iteration 504 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:23:01,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:01,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:01,744][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:01,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:01,778][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:01,778][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:02,485][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:02,780][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:03,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:03,426][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:03,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:04,072][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:04,394][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:04,716][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:05,040][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:05,362][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:05,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:06,011][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:06,336][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:06,661][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:06,985][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:07,309][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:07,632][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:07,954][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:08,277][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:08,599][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:08,924][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:09,247][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:09,570][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:09,894][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:10,221][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:23:10,542][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:23:10,863][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:23:11,188][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:23:11,511][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:23:11,842][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:23:12,163][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:23:12,485][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:23:12,807][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:23:13,498][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:23:14,174][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:23:14,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:23:14,177][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:23:15,130][__main__][INFO] - Iteration 505 took 22s (38.54% Gen, 57.24% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 35m 21s. Estimated total time: 18h 51m 47s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 43s, 500 more iterations: 3h 8m 37s.
[2025-11-13 11:23:15,132][__main__][INFO] - Starting iteration 505.
[2025-11-13 11:23:15,135][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:23:15,136][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:24,070][__main__][INFO] - Number of regex retries in iteration 505: 0
[2025-11-13 11:23:24,070][__main__][INFO] - agents played in iteration 505 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:23:24,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:24,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:24,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:24,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:24,633][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:24,634][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:25,359][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:25,979][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:26,301][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:26,623][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:26,945][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:27,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:27,589][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:27,912][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:28,238][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:28,561][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:28,883][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:29,207][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:29,529][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:29,851][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:30,173][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:30,496][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:30,819][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:31,141][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:31,463][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:31,786][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:32,111][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:32,435][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:32,757][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:33,080][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:23:33,402][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:23:33,724][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:23:34,047][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:23:34,369][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:23:34,692][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:23:35,016][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:23:35,338][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:23:35,664][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:23:36,381][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:23:37,079][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:23:37,080][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:23:37,082][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:23:38,043][__main__][INFO] - Iteration 506 took 22s (38.99% Gen, 56.80% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 48m 37s. Estimated total time: 19h 5m 26s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 10s, 500 more iterations: 3h 10m 54s.
[2025-11-13 11:23:38,045][__main__][INFO] - Starting iteration 506.
[2025-11-13 11:23:38,048][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:23:38,049][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:46,981][__main__][INFO] - Number of regex retries in iteration 506: 0
[2025-11-13 11:23:46,981][__main__][INFO] - agents played in iteration 506 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:23:47,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:47,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:47,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:47,537][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:47,538][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:47,539][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:48,252][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:48,546][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:48,870][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:49,193][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:49,516][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:49,837][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:50,159][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:50,482][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:50,804][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:51,126][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:51,448][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:51,770][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:52,093][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:52,414][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:52,735][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:53,057][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:53,382][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:53,702][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:54,027][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:54,351][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:54,676][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:54,999][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:55,322][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:55,967][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:23:56,293][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:23:56,615][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:23:56,940][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:23:57,268][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:23:57,593][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:23:57,916][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:23:58,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:23:58,562][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:23:59,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:23:59,942][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:23:59,944][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:23:59,946][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:00,937][__main__][INFO] - Iteration 507 took 22s (39.02% Gen, 56.64% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 47m 17s. Estimated total time: 19h 4m 29s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 8s, 500 more iterations: 3h 10m 44s.
[2025-11-13 11:24:00,939][__main__][INFO] - Starting iteration 507.
[2025-11-13 11:24:00,941][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:24:00,942][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:24:09,898][__main__][INFO] - Number of regex retries in iteration 507: 0
[2025-11-13 11:24:09,898][__main__][INFO] - agents played in iteration 507 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:24:10,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:10,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:10,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:10,450][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:10,451][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:24:10,451][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:24:11,161][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:24:11,454][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:24:11,777][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:24:12,100][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:24:12,423][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:24:12,746][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:24:13,070][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:24:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:24:13,716][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:24:14,043][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:24:14,365][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:24:14,689][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:24:15,017][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:24:15,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:24:15,668][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:24:15,996][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:24:16,318][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:24:16,639][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:24:16,965][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:24:17,293][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:24:17,621][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:24:17,950][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:24:18,275][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:24:18,599][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:24:18,924][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:24:19,247][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:24:19,573][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:24:19,897][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:24:20,222][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:24:20,546][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:24:20,870][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:24:21,196][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:24:21,524][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:24:22,237][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:24:22,911][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:24:22,912][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:24:22,913][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:23,868][__main__][INFO] - Iteration 508 took 22s (39.06% Gen, 56.76% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 48m 49s. Estimated total time: 19h 6m 24s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 12s, 500 more iterations: 3h 11m 4s.
[2025-11-13 11:24:23,870][__main__][INFO] - Starting iteration 508.
[2025-11-13 11:24:23,873][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:24:23,873][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:24:32,601][__main__][INFO] - Number of regex retries in iteration 508: 0
[2025-11-13 11:24:32,601][__main__][INFO] - agents played in iteration 508 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:24:33,066][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:33,099][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:33,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:33,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:33,167][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:24:33,168][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:24:33,875][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:24:34,169][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:24:34,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:24:34,817][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:24:35,138][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:24:35,461][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:24:35,783][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:24:36,107][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:24:36,431][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:24:36,755][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:24:37,079][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:24:37,400][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:24:37,727][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:24:38,048][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:24:38,372][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:24:38,696][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:24:39,019][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:24:39,341][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:24:39,664][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:24:39,987][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:24:40,310][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:24:40,633][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:24:40,955][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:24:41,276][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:24:41,600][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:24:41,923][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:24:42,247][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:24:42,571][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:24:42,895][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:24:43,218][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:24:43,540][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:24:43,863][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:24:44,187][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:24:44,868][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:24:45,609][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:24:45,611][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:24:45,612][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:46,535][__main__][INFO] - Iteration 509 took 22s (38.51% Gen, 57.41% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 35m 12s. Estimated total time: 18h 53m 10s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 46s, 500 more iterations: 3h 8m 51s.
[2025-11-13 11:24:46,537][__main__][INFO] - Starting iteration 509.
[2025-11-13 11:24:46,540][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:24:46,541][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:24:55,550][__main__][INFO] - Number of regex retries in iteration 509: 0
[2025-11-13 11:24:55,551][__main__][INFO] - agents played in iteration 509 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:24:56,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:56,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:56,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:56,111][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:56,112][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:24:56,113][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:24:56,819][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:24:57,113][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:24:57,436][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:24:57,757][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:24:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:24:58,402][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:24:58,724][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:24:59,046][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:24:59,368][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:24:59,691][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:00,015][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:00,336][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:00,658][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:00,981][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:01,304][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:01,625][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:01,950][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:02,272][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:02,596][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:02,917][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:03,239][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:03,563][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:03,885][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:04,207][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:04,529][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:04,851][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:05,173][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:05,495][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:05,817][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:06,141][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:06,464][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:06,788][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:07,112][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:07,796][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:25:08,510][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:25:08,511][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:25:08,513][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:25:09,539][__main__][INFO] - Iteration 510 took 22s (39.17% Gen, 56.36% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 51m 37s. Estimated total time: 19h 9m 58s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 19s, 500 more iterations: 3h 11m 39s.
[2025-11-13 11:25:09,541][__main__][INFO] - Starting iteration 510.
[2025-11-13 11:25:09,543][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:25:09,544][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:25:18,794][__main__][INFO] - Number of regex retries in iteration 510: 0
[2025-11-13 11:25:18,795][__main__][INFO] - agents played in iteration 510 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:25:19,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:19,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:19,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:19,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:19,355][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:25:19,355][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:25:20,061][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:25:20,355][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:25:20,678][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:25:21,001][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:25:21,324][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:25:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:25:21,969][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:25:22,291][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:25:22,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:25:22,941][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:23,268][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:23,597][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:23,918][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:24,242][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:24,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:24,887][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:25,209][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:25,531][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:25,856][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:26,177][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:26,505][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:26,828][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:27,150][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:27,474][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:27,796][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:28,120][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:28,442][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:28,763][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:29,084][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:29,408][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:29,731][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:30,053][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:30,378][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:31,064][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:25:31,800][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:25:31,802][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:25:31,804][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:25:33,773][__main__][INFO] - Iteration 511 took 24s (38.18% Gen, 53.69% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 52m 47s. Estimated total time: 20h 11m 32s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 23s, 500 more iterations: 3h 21m 55s.
[2025-11-13 11:25:33,775][__main__][INFO] - Starting iteration 511.
[2025-11-13 11:25:33,778][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:25:33,779][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:25:43,220][__main__][INFO] - Number of regex retries in iteration 511: 0
[2025-11-13 11:25:43,221][__main__][INFO] - agents played in iteration 511 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:25:43,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:43,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:43,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:43,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:43,772][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:25:43,772][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:25:44,486][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:25:44,780][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:25:45,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:25:45,427][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:25:45,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:25:46,071][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:25:46,395][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:25:46,719][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:25:47,041][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:25:47,364][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:47,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:48,016][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:48,339][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:48,662][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:48,986][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:49,308][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:49,636][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:49,963][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:50,284][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:50,606][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:50,928][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:51,252][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:51,577][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:51,904][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:52,226][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:52,550][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:52,873][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:53,199][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:53,527][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:53,855][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:54,179][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:54,503][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:54,827][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:55,537][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:25:56,222][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:25:56,223][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:25:56,225][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:25:57,249][__main__][INFO] - Iteration 512 took 23s (40.23% Gen, 55.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 14m 26s. Estimated total time: 19h 33m 34s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 7s, 500 more iterations: 3h 15m 35s.
[2025-11-13 11:25:57,252][__main__][INFO] - Starting iteration 512.
[2025-11-13 11:25:57,255][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:25:57,256][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:06,471][__main__][INFO] - Number of regex retries in iteration 512: 0
[2025-11-13 11:26:06,471][__main__][INFO] - agents played in iteration 512 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:26:06,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:06,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:06,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:07,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:07,023][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:07,023][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:07,736][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:08,031][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:08,354][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:08,678][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:09,002][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:09,326][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:09,651][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:09,974][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:10,302][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:10,630][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:26:10,955][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:26:11,277][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:26:11,600][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:26:11,922][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:26:12,244][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:26:12,566][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:26:12,889][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:26:13,210][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:26:13,532][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:26:13,855][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:26:14,182][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:26:14,506][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:26:14,829][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:26:15,151][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:26:15,474][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:26:15,798][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:26:16,121][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:26:16,443][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:26:16,765][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:26:17,088][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:26:17,412][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:26:17,736][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:26:18,061][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:26:18,751][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:26:19,434][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:26:19,435][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:26:19,437][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:26:20,310][__main__][INFO] - Iteration 513 took 23s (39.97% Gen, 56.24% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 53m 14s. Estimated total time: 19h 12m 46s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 25s, 500 more iterations: 3h 12m 7s.
[2025-11-13 11:26:20,312][__main__][INFO] - Starting iteration 513.
[2025-11-13 11:26:20,314][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:26:20,315][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:29,791][__main__][INFO] - Number of regex retries in iteration 513: 0
[2025-11-13 11:26:29,792][__main__][INFO] - agents played in iteration 513 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:26:30,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:30,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:30,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:30,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:30,348][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:30,349][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:31,051][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:31,346][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:31,669][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:31,993][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:32,320][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:32,643][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:32,966][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:33,291][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:33,614][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:33,936][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:26:34,259][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:26:34,582][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:26:34,904][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:26:35,227][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:26:35,551][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:26:35,872][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:26:36,194][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:26:36,518][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:26:36,841][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:26:37,164][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:26:37,488][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:26:37,812][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:26:38,134][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:26:38,456][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:26:38,780][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:26:39,104][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:26:39,432][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:26:39,756][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:26:40,082][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:26:40,405][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:26:40,730][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:26:41,053][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:26:41,376][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:26:42,068][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:26:42,750][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:26:42,751][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:26:42,753][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:26:43,859][__main__][INFO] - Iteration 514 took 23s (40.25% Gen, 55.05% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 17m 22s. Estimated total time: 19h 37m 17s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 12s.
[2025-11-13 11:26:43,861][__main__][INFO] - Starting iteration 514.
[2025-11-13 11:26:43,864][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:26:43,864][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:52,891][__main__][INFO] - Number of regex retries in iteration 514: 0
[2025-11-13 11:26:52,891][__main__][INFO] - agents played in iteration 514 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:26:53,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:53,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:53,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:53,448][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:53,448][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:53,449][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:54,170][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:54,788][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:55,110][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:55,432][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:55,756][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:56,078][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:56,401][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:56,725][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:57,052][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:26:57,374][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:26:57,697][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:26:58,018][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:26:58,340][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:26:58,662][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:26:58,983][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:26:59,305][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:26:59,627][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:26:59,950][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:00,272][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:00,598][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:00,922][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:01,247][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:01,570][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:01,892][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:02,214][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:02,538][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:02,862][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:03,185][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:03,509][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:03,832][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:04,155][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:04,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:05,187][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:05,880][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:05,882][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:05,883][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:06,836][__main__][INFO] - Iteration 515 took 22s (39.29% Gen, 56.55% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 48m 20s. Estimated total time: 19h 8m 38s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 17s, 500 more iterations: 3h 11m 26s.
[2025-11-13 11:27:06,838][__main__][INFO] - Starting iteration 515.
[2025-11-13 11:27:06,841][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:06,841][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:27:15,539][__main__][INFO] - Number of regex retries in iteration 515: 0
[2025-11-13 11:27:15,539][__main__][INFO] - agents played in iteration 515 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:27:16,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:16,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:16,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:16,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:16,103][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:27:16,104][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:27:16,812][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:27:17,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:27:17,430][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:27:17,753][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:27:18,075][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:27:18,399][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:27:18,725][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:27:19,048][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:27:19,375][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:27:19,701][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:27:20,029][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:27:20,356][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:27:20,681][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:27:21,004][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:27:21,327][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:27:21,651][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:27:21,974][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:27:22,303][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:27:22,625][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:22,949][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:23,273][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:23,595][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:23,919][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:24,244][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:24,567][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:24,889][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:25,213][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:25,535][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:25,858][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:26,181][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:26,505][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:26,828][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:27,153][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:27,859][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:28,545][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:28,547][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:28,548][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:29,496][__main__][INFO] - Iteration 516 took 22s (38.39% Gen, 57.42% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 32m 8s. Estimated total time: 18h 52m 48s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 45s, 500 more iterations: 3h 8m 48s.
[2025-11-13 11:27:29,498][__main__][INFO] - Starting iteration 516.
[2025-11-13 11:27:29,500][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:29,501][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:27:38,428][__main__][INFO] - Number of regex retries in iteration 516: 0
[2025-11-13 11:27:38,429][__main__][INFO] - agents played in iteration 516 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:27:38,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:38,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:38,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:38,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:38,978][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:27:38,979][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:27:39,701][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:27:39,996][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:27:40,319][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:27:40,641][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:27:40,965][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:27:41,289][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:27:41,612][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:27:41,935][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:27:42,259][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:27:42,582][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:27:42,904][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:27:43,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:27:43,554][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:27:43,879][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:27:44,202][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:27:44,526][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:27:44,849][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:27:45,176][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:27:45,502][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:45,825][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:46,149][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:46,477][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:46,802][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:47,125][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:47,772][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:48,095][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:48,418][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:48,741][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:49,065][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:49,387][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:49,709][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:50,033][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:50,744][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:51,436][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:51,438][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:51,439][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:52,412][__main__][INFO] - Iteration 517 took 22s (38.96% Gen, 56.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 44m 35s. Estimated total time: 19h 5m 38s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 56s.
[2025-11-13 11:27:52,414][__main__][INFO] - Starting iteration 517.
[2025-11-13 11:27:52,418][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:52,418][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:01,030][__main__][INFO] - Number of regex retries in iteration 517: 0
[2025-11-13 11:28:01,031][__main__][INFO] - agents played in iteration 517 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:28:01,483][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:01,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:01,551][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:01,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:01,585][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:01,585][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:02,296][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:02,591][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:02,914][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:03,235][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:03,559][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:03,881][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:04,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:04,528][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:04,852][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:05,175][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:05,501][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:05,826][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:06,155][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:06,477][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:06,802][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:07,128][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:07,451][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:07,776][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:08,106][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:08,436][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:08,762][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:28:09,085][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:28:09,414][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:28:09,740][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:28:10,066][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:28:10,388][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:28:10,711][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:28:11,035][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:28:11,358][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:28:11,681][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:28:12,004][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:28:12,328][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:28:12,654][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:28:13,350][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:28:14,034][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:28:14,036][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:28:14,038][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:28:14,864][__main__][INFO] - Iteration 518 took 22s (38.37% Gen, 57.95% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 20m 55s. Estimated total time: 18h 42m 21s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 24s, 500 more iterations: 3h 7m 3s.
[2025-11-13 11:28:14,867][__main__][INFO] - Starting iteration 518.
[2025-11-13 11:28:14,871][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:28:14,871][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:24,082][__main__][INFO] - Number of regex retries in iteration 518: 0
[2025-11-13 11:28:24,082][__main__][INFO] - agents played in iteration 518 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:28:24,546][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:24,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:24,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:24,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:24,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:24,649][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:25,357][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:25,651][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:25,974][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:26,297][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:26,621][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:26,947][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:27,269][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:27,593][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:27,916][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:28,239][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:28,567][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:28,889][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:29,215][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:29,538][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:29,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:30,190][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:30,518][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:30,846][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:31,169][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:31,493][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:31,817][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:28:32,146][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:28:32,472][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:28:32,796][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:28:33,121][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:28:33,444][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:28:33,770][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:28:34,092][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:28:34,415][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:28:34,738][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:28:35,062][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:28:35,388][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:28:35,712][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:28:36,404][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:28:37,116][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:28:37,119][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:28:37,120][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:28:38,105][__main__][INFO] - Iteration 519 took 23s (39.64% Gen, 56.11% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 59m 56s. Estimated total time: 19h 21m 45s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 37s.
[2025-11-13 11:28:38,107][__main__][INFO] - Starting iteration 519.
[2025-11-13 11:28:38,111][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:28:38,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:46,809][__main__][INFO] - Number of regex retries in iteration 519: 0
[2025-11-13 11:28:46,810][__main__][INFO] - agents played in iteration 519 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:28:47,258][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:47,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:47,325][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:47,359][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:47,359][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:47,359][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:48,085][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:48,379][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:48,703][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:49,349][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:49,673][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:49,996][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:50,319][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:50,643][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:50,966][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:51,290][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:51,613][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:51,938][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:52,261][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:52,585][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:52,909][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:53,558][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:53,883][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:54,204][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:54,528][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:28:54,856][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:28:55,179][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:28:55,505][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:28:55,834][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:28:56,157][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:28:56,485][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:28:56,808][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:28:57,132][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:28:57,454][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:28:57,777][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:28:58,100][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:28:58,422][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:28:59,116][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:28:59,799][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:28:59,801][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:28:59,802][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:00,752][__main__][INFO] - Iteration 520 took 22s (38.42% Gen, 57.38% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 29m 55s. Estimated total time: 18h 52m 6s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 44s, 500 more iterations: 3h 8m 41s.
[2025-11-13 11:29:00,754][__main__][INFO] - Starting iteration 520.
[2025-11-13 11:29:00,757][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:29:00,758][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:29:09,738][__main__][INFO] - Number of regex retries in iteration 520: 0
[2025-11-13 11:29:09,738][__main__][INFO] - agents played in iteration 520 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:29:10,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:10,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:10,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:10,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:10,305][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:29:10,305][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:29:11,025][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:29:11,320][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:29:11,643][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:29:11,968][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:29:12,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:29:12,613][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:29:12,937][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:29:13,260][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:29:13,583][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:29:13,906][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:29:14,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:29:14,553][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:29:14,878][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:29:15,205][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:29:15,531][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:29:15,858][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:29:16,181][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:29:16,503][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:29:16,825][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:29:17,147][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:29:17,473][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:29:17,794][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:29:18,118][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:29:18,443][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:29:18,768][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:29:19,090][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:29:19,413][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:29:19,737][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:29:20,060][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:29:20,383][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:29:20,708][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:29:21,031][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:29:21,356][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:29:22,043][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:29:22,727][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:29:22,730][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:29:22,732][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:24,900][__main__][INFO] - Iteration 521 took 24s (37.19% Gen, 53.82% Train). Generation: 8s, Training: 12s. Estimated remaining time: 19h 44m 38s. Estimated total time: 20h 7m 13s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 14s, 500 more iterations: 3h 21m 12s.
[2025-11-13 11:29:24,902][__main__][INFO] - Starting iteration 521.
[2025-11-13 11:29:24,906][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:29:24,906][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:29:34,098][__main__][INFO] - Number of regex retries in iteration 521: 0
[2025-11-13 11:29:34,098][__main__][INFO] - agents played in iteration 521 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:29:34,576][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:34,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:34,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:34,677][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:34,677][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:29:34,678][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:29:35,387][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:29:35,681][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:29:36,004][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:29:36,327][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:29:36,652][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:29:36,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:29:37,299][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:29:37,623][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:29:37,948][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:29:38,271][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:29:38,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:29:38,916][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:29:39,239][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:29:39,564][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:29:39,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:29:40,212][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:29:40,535][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:29:40,858][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:29:41,183][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:29:41,508][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:29:41,832][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:29:42,155][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:29:42,479][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:29:42,803][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:29:43,126][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:29:43,450][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:29:43,773][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:29:44,096][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:29:44,420][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:29:44,742][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:29:45,065][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:29:45,389][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:29:45,713][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:29:46,390][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:29:47,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:29:47,106][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:29:47,107][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:48,224][__main__][INFO] - Iteration 522 took 23s (39.42% Gen, 55.78% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 2m 59s. Estimated total time: 19h 25m 58s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 51s, 500 more iterations: 3h 14m 19s.
[2025-11-13 11:29:48,226][__main__][INFO] - Starting iteration 522.
[2025-11-13 11:29:48,230][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:29:48,230][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:29:57,292][__main__][INFO] - Number of regex retries in iteration 522: 0
[2025-11-13 11:29:57,292][__main__][INFO] - agents played in iteration 522 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:29:57,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:57,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:57,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:57,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:57,852][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:29:57,852][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:29:58,563][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:29:58,857][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:29:59,181][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:29:59,503][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:29:59,825][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:00,148][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:00,472][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:00,795][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:01,118][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:01,443][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:01,766][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:02,091][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:02,414][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:02,739][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:03,068][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:03,393][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:03,717][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:04,044][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:04,368][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:04,692][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:05,015][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:05,339][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:05,662][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:05,985][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:06,309][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:06,632][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:06,956][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:07,928][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:08,253][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:08,581][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:08,906][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:09,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:10,293][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:10,294][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:10,296][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:30:11,153][__main__][INFO] - Iteration 523 took 22s (39.53% Gen, 56.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 42m 49s. Estimated total time: 19h 6m 11s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 12s, 500 more iterations: 3h 11m 1s.
[2025-11-13 11:30:11,155][__main__][INFO] - Starting iteration 523.
[2025-11-13 11:30:11,158][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:30:11,159][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:30:20,606][__main__][INFO] - Number of regex retries in iteration 523: 0
[2025-11-13 11:30:20,606][__main__][INFO] - agents played in iteration 523 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:30:21,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:21,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:21,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:21,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:21,166][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:30:21,167][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:30:21,877][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:30:22,170][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:30:22,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:30:22,816][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:30:23,139][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:23,461][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:23,785][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:24,109][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:24,432][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:24,757][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:25,083][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:25,406][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:25,730][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:26,054][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:26,381][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:26,705][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:27,028][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:27,350][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:27,673][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:27,997][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:28,321][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:28,643][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:28,967][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:29,291][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:29,614][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:29,939][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:30,262][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:30,587][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:30,909][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:31,233][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:31,556][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:31,879][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:32,203][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:32,876][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:33,586][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:33,588][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:33,589][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:30:34,591][__main__][INFO] - Iteration 524 took 23s (40.32% Gen, 55.40% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 7m 54s. Estimated total time: 19h 31m 40s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 3s, 500 more iterations: 3h 15m 16s.
[2025-11-13 11:30:34,593][__main__][INFO] - Starting iteration 524.
[2025-11-13 11:30:34,596][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:30:34,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:30:43,755][__main__][INFO] - Number of regex retries in iteration 524: 0
[2025-11-13 11:30:43,755][__main__][INFO] - agents played in iteration 524 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:30:44,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:44,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:44,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:44,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:44,309][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:30:44,309][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:30:45,016][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:30:45,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:30:45,634][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:30:45,957][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:30:46,284][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:46,607][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:46,931][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:47,255][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:47,584][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:47,907][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:48,230][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:48,552][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:48,874][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:49,199][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:49,526][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:49,856][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:50,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:50,503][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:50,825][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:51,149][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:51,472][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:51,798][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:52,121][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:52,446][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:52,772][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:53,096][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:53,418][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:53,743][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:54,066][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:54,391][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:54,714][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:55,038][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:55,363][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:56,060][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:56,775][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:56,777][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:56,778][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:30:57,762][__main__][INFO] - Iteration 525 took 23s (39.53% Gen, 56.21% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 12s. Estimated total time: 19h 18m 21s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 3s.
[2025-11-13 11:30:57,764][__main__][INFO] - Starting iteration 525.
[2025-11-13 11:30:57,767][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:30:57,768][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:06,890][__main__][INFO] - Number of regex retries in iteration 525: 0
[2025-11-13 11:31:06,891][__main__][INFO] - agents played in iteration 525 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:31:07,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:07,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:07,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:07,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:07,448][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:07,448][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:08,155][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:08,450][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:08,773][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:09,097][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:09,422][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:09,744][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:10,067][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:10,392][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:10,716][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:11,044][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:31:11,368][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:31:11,691][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:31:12,014][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:31:12,338][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:31:12,662][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:31:12,985][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:31:13,309][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:31:13,632][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:31:13,955][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:31:14,279][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:31:14,602][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:31:14,925][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:31:15,248][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:31:15,573][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:31:15,898][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:31:16,223][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:31:16,546][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:31:16,869][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:31:17,194][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:31:17,518][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:31:17,840][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:31:18,167][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:31:18,492][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:31:19,184][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:31:19,913][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:31:19,915][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:31:19,917][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:31:20,920][__main__][INFO] - Iteration 526 took 23s (39.40% Gen, 56.26% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 53m 8s. Estimated total time: 19h 17m 40s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 56s.
[2025-11-13 11:31:20,922][__main__][INFO] - Starting iteration 526.
[2025-11-13 11:31:20,926][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:31:20,927][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:29,668][__main__][INFO] - Number of regex retries in iteration 526: 0
[2025-11-13 11:31:29,669][__main__][INFO] - agents played in iteration 526 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:31:30,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:30,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:30,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:30,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:30,224][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:30,224][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:30,942][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:31,238][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:31,560][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:31,883][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:32,208][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:32,530][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:32,853][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:33,177][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:33,501][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:31:34,147][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:31:34,470][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:31:34,799][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:31:35,124][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:31:35,449][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:31:35,772][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:31:36,096][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:31:36,419][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:31:36,741][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:31:37,064][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:31:37,387][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:31:37,710][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:31:38,035][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:31:38,360][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:31:38,683][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:31:39,006][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:31:39,329][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:31:39,652][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:31:39,976][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:31:40,299][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:31:40,627][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:31:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:31:41,279][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:31:41,962][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:31:42,681][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:31:42,683][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:31:42,685][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:31:43,685][__main__][INFO] - Iteration 527 took 22s (38.41% Gen, 57.18% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 33m 4s. Estimated total time: 18h 57m 59s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 55s, 500 more iterations: 3h 9m 39s.
[2025-11-13 11:31:43,687][__main__][INFO] - Starting iteration 527.
[2025-11-13 11:31:43,690][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:31:43,691][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:52,804][__main__][INFO] - Number of regex retries in iteration 527: 0
[2025-11-13 11:31:52,805][__main__][INFO] - agents played in iteration 527 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:31:53,272][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:53,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:53,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:53,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:53,374][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:53,374][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:54,082][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:54,377][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:54,700][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:55,022][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:55,345][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:55,668][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:55,995][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:56,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:56,645][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:56,969][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:31:57,292][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:31:57,617][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:31:57,941][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:31:58,264][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:31:58,588][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:31:58,911][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:31:59,234][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:31:59,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:31:59,880][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:00,202][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:00,524][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:00,847][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:01,172][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:01,495][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:01,818][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:02,146][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:02,470][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:02,794][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:03,118][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:03,441][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:03,762][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:04,087][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:04,410][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:05,086][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:05,802][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:05,803][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:05,805][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:06,865][__main__][INFO] - Iteration 528 took 23s (39.33% Gen, 56.10% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 53m 29s. Estimated total time: 19h 18m 47s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 7s.
[2025-11-13 11:32:06,867][__main__][INFO] - Starting iteration 528.
[2025-11-13 11:32:06,871][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:32:06,871][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:32:16,095][__main__][INFO] - Number of regex retries in iteration 528: 0
[2025-11-13 11:32:16,096][__main__][INFO] - agents played in iteration 528 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:32:16,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:16,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:16,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:16,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:16,652][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:32:16,653][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:32:17,364][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:32:17,658][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:32:17,982][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:32:18,306][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:32:18,629][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:32:18,954][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:32:19,277][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:32:19,601][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:32:19,925][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:32:20,249][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:32:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:32:20,897][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:32:21,219][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:32:21,542][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:32:21,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:32:22,188][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:32:22,512][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:32:22,835][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:32:23,158][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:23,481][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:23,803][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:24,127][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:24,450][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:24,773][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:25,097][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:25,421][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:25,745][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:26,071][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:26,396][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:26,720][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:27,042][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:27,365][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:27,687][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:28,362][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:29,075][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:29,077][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:29,078][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:30,167][__main__][INFO] - Iteration 529 took 23s (39.59% Gen, 55.73% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 59m 9s. Estimated total time: 19h 24m 50s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 49s, 500 more iterations: 3h 14m 8s.
[2025-11-13 11:32:30,169][__main__][INFO] - Starting iteration 529.
[2025-11-13 11:32:30,172][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:32:30,172][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:32:38,822][__main__][INFO] - Number of regex retries in iteration 529: 0
[2025-11-13 11:32:38,822][__main__][INFO] - agents played in iteration 529 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:32:39,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:39,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:39,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:39,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:39,376][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:32:39,377][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:32:40,091][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:32:40,386][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:32:40,710][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:32:41,032][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:32:41,358][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:32:41,682][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:32:42,005][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:32:42,328][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:32:42,652][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:32:42,977][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:32:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:32:43,625][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:32:43,947][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:32:44,271][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:32:44,594][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:32:44,917][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:32:45,240][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:32:45,568][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:32:45,891][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:46,217][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:46,541][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:46,865][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:47,189][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:47,514][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:47,837][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:48,162][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:48,489][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:48,811][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:49,134][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:49,457][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:49,781][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:50,104][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:50,426][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:51,102][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:51,817][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:51,819][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:51,821][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:52,856][__main__][INFO] - Iteration 530 took 22s (38.13% Gen, 57.30% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 28m 12s. Estimated total time: 18h 54m 15s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 48s, 500 more iterations: 3h 9m 2s.
[2025-11-13 11:32:52,858][__main__][INFO] - Starting iteration 530.
[2025-11-13 11:32:52,861][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:32:52,862][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:01,461][__main__][INFO] - Number of regex retries in iteration 530: 0
[2025-11-13 11:33:01,461][__main__][INFO] - agents played in iteration 530 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:33:01,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:01,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:01,985][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:02,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:02,020][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:02,020][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:02,728][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:03,023][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:03,346][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:03,669][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:03,992][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:04,317][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:04,642][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:04,966][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:05,288][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:05,611][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:05,936][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:06,258][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:06,582][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:06,907][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:07,556][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:07,885][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:08,208][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:08,531][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:08,855][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:09,183][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:09,506][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:09,829][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:10,152][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:10,477][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:10,802][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:11,127][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:11,452][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:33:11,781][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:33:12,104][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:33:12,426][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:33:12,749][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:33:13,074][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:33:13,752][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:33:14,473][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:33:14,475][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:33:14,477][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:33:16,452][__main__][INFO] - Iteration 531 took 23s (36.45% Gen, 55.17% Train). Generation: 8s, Training: 13s. Estimated remaining time: 19h 13m 6s. Estimated total time: 19h 39m 34s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 35s.
[2025-11-13 11:33:16,454][__main__][INFO] - Starting iteration 531.
[2025-11-13 11:33:16,458][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:33:16,458][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:25,882][__main__][INFO] - Number of regex retries in iteration 531: 0
[2025-11-13 11:33:25,882][__main__][INFO] - agents played in iteration 531 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:33:26,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:26,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:26,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:26,439][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:26,440][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:26,440][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:27,171][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:27,467][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:27,791][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:28,116][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:28,440][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:28,762][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:29,086][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:29,409][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:29,731][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:30,054][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:30,379][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:30,702][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:31,025][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:31,348][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:31,671][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:31,996][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:32,319][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:32,643][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:32,966][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:33,290][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:33,614][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:34,262][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:34,589][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:34,912][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:35,235][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:35,562][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:35,887][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:33:36,212][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:33:36,536][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:33:36,859][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:33:37,184][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:33:37,513][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:33:38,205][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:33:38,917][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:33:38,919][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:33:38,920][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:33:40,058][__main__][INFO] - Iteration 532 took 23s (39.93% Gen, 55.24% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 13m 13s. Estimated total time: 19h 40m 4s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 20s, 500 more iterations: 3h 16m 40s.
[2025-11-13 11:33:40,060][__main__][INFO] - Starting iteration 532.
[2025-11-13 11:33:40,064][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:33:40,064][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:49,495][__main__][INFO] - Number of regex retries in iteration 532: 0
[2025-11-13 11:33:49,496][__main__][INFO] - agents played in iteration 532 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:33:49,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:49,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,062][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:50,063][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:50,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:51,088][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:51,412][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:51,740][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:52,068][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:52,391][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:52,715][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:53,038][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:53,361][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:53,685][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:54,008][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:54,330][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:54,653][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:54,975][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:55,301][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:55,624][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:55,946][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:56,276][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:56,601][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:56,925][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:57,250][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:57,575][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:57,898][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:58,222][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:58,545][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:58,867][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:59,191][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:59,514][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:33:59,838][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:00,162][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:00,485][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:00,807][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:01,129][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:01,824][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:02,545][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:02,547][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:02,549][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:03,687][__main__][INFO] - Iteration 533 took 23s (39.92% Gen, 55.25% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 13m 58s. Estimated total time: 19h 41m 13s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 22s, 500 more iterations: 3h 16m 52s.
[2025-11-13 11:34:03,689][__main__][INFO] - Starting iteration 533.
[2025-11-13 11:34:03,692][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:03,692][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:13,089][__main__][INFO] - Number of regex retries in iteration 533: 0
[2025-11-13 11:34:13,090][__main__][INFO] - agents played in iteration 533 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:34:13,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,657][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:13,657][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:34:14,383][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:34:14,678][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:34:15,002][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:34:15,325][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:34:15,649][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:34:15,971][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:34:16,293][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:34:16,617][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:34:16,942][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:34:17,265][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:34:17,590][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:34:17,913][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:34:18,237][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:34:18,560][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:34:18,884][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:34:19,207][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:34:19,530][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:34:19,853][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:34:20,182][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:34:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:34:20,835][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:34:21,163][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:34:21,488][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:34:21,813][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:34:22,135][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:34:22,458][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:34:22,782][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:34:23,104][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:34:23,429][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:23,754][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:24,078][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:24,400][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:24,724][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:25,406][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:26,131][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:26,132][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:26,134][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:27,154][__main__][INFO] - Iteration 534 took 23s (40.05% Gen, 55.59% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 5m 29s. Estimated total time: 19h 33m 8s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 6s, 500 more iterations: 3h 15m 31s.
[2025-11-13 11:34:27,156][__main__][INFO] - Starting iteration 534.
[2025-11-13 11:34:27,159][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:27,159][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:36,204][__main__][INFO] - Number of regex retries in iteration 534: 0
[2025-11-13 11:34:36,205][__main__][INFO] - agents played in iteration 534 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:34:36,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,691][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,760][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:36,760][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:34:37,488][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:34:37,782][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:34:38,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:34:38,430][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:34:38,754][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:34:39,077][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:34:39,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:34:39,724][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:34:40,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:34:40,369][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:34:40,693][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:34:41,017][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:34:41,341][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:34:41,665][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:34:41,989][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:34:42,313][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:34:42,637][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:34:42,962][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:34:43,286][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:34:43,608][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:34:43,934][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:34:44,257][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:34:44,580][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:34:44,903][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:34:45,226][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:34:45,550][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:34:45,872][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:34:46,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:34:46,521][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:46,844][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:47,171][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:47,497][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:47,819][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:48,509][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:49,254][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:49,255][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:49,257][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:50,201][__main__][INFO] - Iteration 535 took 23s (39.25% Gen, 56.64% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 44m 7s. Estimated total time: 19h 12m 8s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 24s, 500 more iterations: 3h 12m 1s.
[2025-11-13 11:34:50,203][__main__][INFO] - Starting iteration 535.
[2025-11-13 11:34:50,206][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:50,207][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:59,307][__main__][INFO] - Number of regex retries in iteration 535: 0
[2025-11-13 11:34:59,308][__main__][INFO] - agents played in iteration 535 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:34:59,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,870][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,871][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:59,871][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:00,601][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:00,896][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:01,219][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:01,543][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:01,866][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:02,188][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:02,511][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:02,835][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:03,158][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:03,482][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:03,805][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:04,128][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:04,453][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:04,776][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:05,098][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:05,422][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:05,744][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:06,067][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:06,396][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:06,725][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:07,048][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:07,375][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:07,705][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:08,028][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:08,350][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:08,674][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:09,000][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:09,325][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:09,651][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:09,976][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:10,300][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:10,622][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:10,945][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:11,654][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:12,379][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:12,380][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:12,382][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:35:13,421][__main__][INFO] - Iteration 536 took 23s (39.20% Gen, 56.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 52m 22s. Estimated total time: 19h 20m 47s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 41s, 500 more iterations: 3h 13m 27s.
[2025-11-13 11:35:13,423][__main__][INFO] - Starting iteration 536.
[2025-11-13 11:35:13,427][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:35:13,428][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:35:22,964][__main__][INFO] - Number of regex retries in iteration 536: 0
[2025-11-13 11:35:22,964][__main__][INFO] - agents played in iteration 536 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:35:23,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,536][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,536][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:35:23,537][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:24,248][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:24,543][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:24,866][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:25,190][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:25,515][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:25,839][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:26,163][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:26,486][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:26,809][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:27,132][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:27,455][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:27,778][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:28,102][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:28,425][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:28,748][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:29,071][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:29,395][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:29,719][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:30,043][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:30,366][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:30,690][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:31,012][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:31,336][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:31,659][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:31,983][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:32,308][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:32,633][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:32,955][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:33,277][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:33,600][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:33,923][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:34,245][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:34,571][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:35,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:35,986][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:35,987][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:35,989][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:35:36,982][__main__][INFO] - Iteration 537 took 23s (40.48% Gen, 55.29% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 8m 59s. Estimated total time: 19h 37m 47s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 17s.
[2025-11-13 11:35:36,983][__main__][INFO] - Starting iteration 537.
[2025-11-13 11:35:36,987][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:35:36,987][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:35:45,933][__main__][INFO] - Number of regex retries in iteration 537: 0
[2025-11-13 11:35:45,933][__main__][INFO] - agents played in iteration 537 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:35:46,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,490][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:35:46,491][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:47,205][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:47,501][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:47,824][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:48,147][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:48,472][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:48,795][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:49,119][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:49,442][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:49,766][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:50,088][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:50,412][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:50,736][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:51,062][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:51,384][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:51,707][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:52,030][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:52,352][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:52,676][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:52,998][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:53,320][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:53,644][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:53,969][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:54,293][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:54,617][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:54,940][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:55,265][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:55,589][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:55,911][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:56,237][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:56,559][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:56,881][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:57,205][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:57,527][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:58,202][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:58,918][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:58,920][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:58,921][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:35:59,944][__main__][INFO] - Iteration 538 took 22s (38.97% Gen, 56.57% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 38m 43s. Estimated total time: 19h 7m 54s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 19s.
[2025-11-13 11:35:59,946][__main__][INFO] - Starting iteration 538.
[2025-11-13 11:35:59,949][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:35:59,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:09,031][__main__][INFO] - Number of regex retries in iteration 538: 0
[2025-11-13 11:36:09,032][__main__][INFO] - agents played in iteration 538 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:36:09,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:09,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:09,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:09,586][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:09,587][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:09,587][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:10,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:10,613][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:10,936][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:11,260][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:11,584][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:36:11,907][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:36:12,230][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:36:12,552][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:36:12,876][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:36:13,200][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:36:13,523][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:36:13,847][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:36:14,171][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:36:14,494][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:36:14,818][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:36:15,140][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:36:15,464][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:36:15,788][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:36:16,110][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:36:16,432][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:36:16,754][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:36:17,077][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:36:17,401][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:36:17,726][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:36:18,055][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:36:18,379][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:36:18,704][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:36:19,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:36:19,357][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:36:19,678][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:36:20,002][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:36:20,324][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:36:20,647][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:36:21,318][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:36:22,044][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:36:22,045][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:36:22,047][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:36:23,058][__main__][INFO] - Iteration 539 took 23s (39.30% Gen, 56.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 45m 55s. Estimated total time: 19h 15m 29s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 34s.
[2025-11-13 11:36:23,060][__main__][INFO] - Starting iteration 539.
[2025-11-13 11:36:23,064][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:36:23,064][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:31,818][__main__][INFO] - Number of regex retries in iteration 539: 0
[2025-11-13 11:36:31,819][__main__][INFO] - agents played in iteration 539 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:36:32,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:32,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:32,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:32,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:32,380][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:32,381][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:33,100][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:33,396][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:33,722][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:34,045][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:34,368][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:36:34,691][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:36:35,014][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:36:35,337][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:36:35,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:36:35,984][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:36:36,307][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:36:36,631][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:36:36,955][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:36:37,278][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:36:37,607][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:36:37,929][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:36:38,255][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:36:38,578][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:36:38,903][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:36:39,227][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:36:39,554][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:36:39,877][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:36:40,201][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:36:40,528][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:36:40,853][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:36:41,177][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:36:41,503][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:36:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:36:42,154][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:36:42,482][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:36:42,804][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:36:43,128][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:36:43,456][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:36:44,125][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:36:44,841][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:36:44,843][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:36:44,844][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:36:45,863][__main__][INFO] - Iteration 540 took 22s (38.39% Gen, 57.13% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 30m 4s. Estimated total time: 19h 0m 0s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 0s, 500 more iterations: 3h 10m 0s.
[2025-11-13 11:36:45,865][__main__][INFO] - Starting iteration 540.
[2025-11-13 11:36:45,868][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:36:45,869][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:55,254][__main__][INFO] - Number of regex retries in iteration 540: 0
[2025-11-13 11:36:55,254][__main__][INFO] - agents played in iteration 540 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:36:55,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:55,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:55,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:55,819][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:55,820][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:55,820][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:56,531][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:56,827][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:57,151][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:57,474][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:57,798][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:36:58,121][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:36:58,444][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:36:58,767][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:36:59,090][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:36:59,414][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:36:59,738][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:00,062][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:00,386][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:00,709][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:01,033][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:01,358][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:01,682][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:02,003][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:02,328][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:02,650][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:02,974][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:03,296][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:03,620][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:03,944][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:04,271][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:04,597][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:04,921][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:05,244][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:05,568][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:05,890][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:06,213][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:06,537][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:06,867][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:07,527][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:08,245][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:08,246][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:08,248][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:10,247][__main__][INFO] - Iteration 541 took 24s (38.50% Gen, 53.29% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 48m 38s. Estimated total time: 20h 19m 0s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 38s, 500 more iterations: 3h 23m 10s.
[2025-11-13 11:37:10,249][__main__][INFO] - Starting iteration 541.
[2025-11-13 11:37:10,252][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:10,253][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:37:19,791][__main__][INFO] - Number of regex retries in iteration 541: 0
[2025-11-13 11:37:19,792][__main__][INFO] - agents played in iteration 541 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:37:20,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,347][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:37:20,348][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:37:21,068][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:37:21,363][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:37:21,686][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:37:22,009][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:37:22,331][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:37:22,654][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:37:22,978][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:37:23,302][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:37:23,626][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:37:23,950][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:37:24,274][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:24,597][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:24,920][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:25,565][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:25,890][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:26,212][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:26,537][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:26,867][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:27,191][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:27,515][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:27,840][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:28,165][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:28,489][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:28,812][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:29,136][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:29,464][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:29,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:30,114][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:30,437][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:30,761][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:31,084][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:31,405][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:32,086][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:32,804][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:32,805][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:32,807][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:33,819][__main__][INFO] - Iteration 542 took 23s (40.47% Gen, 55.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 39s. Estimated total time: 19h 38m 24s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 24s.
[2025-11-13 11:37:33,826][__main__][INFO] - Starting iteration 542.
[2025-11-13 11:37:33,830][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:33,831][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:37:43,721][__main__][INFO] - Number of regex retries in iteration 542: 0
[2025-11-13 11:37:43,721][__main__][INFO] - agents played in iteration 542 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:37:44,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:44,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:44,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:44,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:44,286][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:37:44,286][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:37:44,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:37:45,291][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:37:45,615][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:37:45,940][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:37:46,263][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:37:46,587][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:37:46,909][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:37:47,232][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:37:47,555][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:37:47,878][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:37:48,205][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:48,529][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:48,852][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:49,176][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:49,503][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:49,825][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:50,148][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:50,472][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:50,796][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:51,123][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:51,446][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:51,769][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:52,091][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:52,415][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:52,739][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:53,064][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:53,387][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:53,710][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:54,032][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:54,360][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:54,685][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:55,010][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:55,334][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:56,033][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:56,754][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:56,755][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:56,757][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:57,753][__main__][INFO] - Iteration 543 took 23s (41.34% Gen, 54.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 25m 5s. Estimated total time: 19h 56m 13s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 52s, 500 more iterations: 3h 19m 22s.
[2025-11-13 11:37:57,755][__main__][INFO] - Starting iteration 543.
[2025-11-13 11:37:57,758][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:57,759][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:06,941][__main__][INFO] - Number of regex retries in iteration 543: 0
[2025-11-13 11:38:06,941][__main__][INFO] - agents played in iteration 543 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:38:07,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,502][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:07,503][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:08,225][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:08,520][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:08,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:09,169][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:09,493][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:09,816][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:10,140][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:10,464][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:10,787][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:11,112][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:11,438][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:11,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:12,086][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:12,415][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:12,741][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:13,066][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:13,393][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:38:13,720][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:38:14,047][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:38:14,374][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:38:14,700][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:38:15,024][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:38:15,347][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:38:15,670][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:38:15,994][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:38:16,319][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:38:16,648][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:38:16,970][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:38:17,297][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:38:17,622][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:38:17,947][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:38:18,276][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:38:18,602][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:38:19,314][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:38:20,059][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:38:20,060][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:38:20,062][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:38:21,032][__main__][INFO] - Iteration 544 took 23s (39.45% Gen, 56.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 52m 14s. Estimated total time: 19h 23m 46s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 57s.
[2025-11-13 11:38:21,034][__main__][INFO] - Starting iteration 544.
[2025-11-13 11:38:21,037][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:38:21,038][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:30,131][__main__][INFO] - Number of regex retries in iteration 544: 0
[2025-11-13 11:38:30,131][__main__][INFO] - agents played in iteration 544 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:38:30,587][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,689][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:30,690][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:31,422][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:31,716][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:32,040][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:32,364][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:32,687][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:33,011][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:33,659][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:33,983][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:34,309][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:34,632][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:34,957][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:35,281][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:35,603][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:35,928][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:36,251][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:36,573][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:38:36,897][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:38:37,221][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:38:37,549][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:38:37,875][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:38:38,198][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:38:38,521][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:38:38,845][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:38:39,166][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:38:39,489][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:38:39,811][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:38:40,133][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:38:40,457][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:38:40,781][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:38:41,104][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:38:41,429][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:38:41,754][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:38:42,445][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:38:43,162][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:38:43,163][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:38:43,165][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:38:44,121][__main__][INFO] - Iteration 545 took 23s (39.39% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 42m 18s. Estimated total time: 19h 14m 13s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 22s.
[2025-11-13 11:38:44,123][__main__][INFO] - Starting iteration 545.
[2025-11-13 11:38:44,126][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:38:44,126][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:53,252][__main__][INFO] - Number of regex retries in iteration 545: 0
[2025-11-13 11:38:53,252][__main__][INFO] - agents played in iteration 545 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:38:53,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,807][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:53,808][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:54,527][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:54,820][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:55,144][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:55,468][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:55,791][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:56,114][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:56,438][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:57,084][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:57,407][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:57,731][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:58,056][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:58,384][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:58,707][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:59,032][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:59,358][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:59,684][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:39:00,009][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:00,338][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:00,661][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:00,985][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:01,315][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:01,639][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:01,960][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:02,285][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:02,608][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:02,930][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:03,259][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:03,587][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:03,912][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:04,237][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:04,560][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:04,882][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:05,570][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:06,282][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:06,284][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:06,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:07,508][__main__][INFO] - Iteration 546 took 23s (39.03% Gen, 55.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 56m 52s. Estimated total time: 19h 29m 10s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 51s.
[2025-11-13 11:39:07,510][__main__][INFO] - Starting iteration 546.
[2025-11-13 11:39:07,513][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:39:07,513][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:39:16,366][__main__][INFO] - Number of regex retries in iteration 546: 0
[2025-11-13 11:39:16,366][__main__][INFO] - agents played in iteration 546 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:39:16,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,929][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:39:16,929][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:39:17,649][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:39:17,943][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:39:18,268][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:39:18,591][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:39:18,914][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:39:19,237][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:39:19,562][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:39:19,884][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:39:20,207][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:39:20,530][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:39:20,854][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:39:21,178][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:39:21,500][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:39:21,823][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:39:22,149][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:39:22,474][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:39:22,798][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:39:23,122][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:23,445][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:23,770][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:24,096][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:24,418][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:24,742][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:25,064][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:25,386][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:25,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:26,032][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:26,356][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:26,678][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:27,002][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:27,326][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:27,649][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:27,972][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:28,672][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:29,402][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:29,403][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:29,405][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:30,396][__main__][INFO] - Iteration 547 took 22s (38.68% Gen, 56.98% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 31m 30s. Estimated total time: 19h 4m 11s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 8s, 500 more iterations: 3h 10m 41s.
[2025-11-13 11:39:30,398][__main__][INFO] - Starting iteration 547.
[2025-11-13 11:39:30,401][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:39:30,401][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:39:40,031][__main__][INFO] - Number of regex retries in iteration 547: 0
[2025-11-13 11:39:40,031][__main__][INFO] - agents played in iteration 547 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:39:40,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,589][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:39:40,589][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:39:41,310][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:39:41,605][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:39:41,930][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:39:42,254][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:39:42,578][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:39:42,901][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:39:43,225][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:39:43,550][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:39:43,875][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:39:44,199][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:39:44,524][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:39:44,848][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:39:45,174][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:39:45,498][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:39:45,823][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:39:46,144][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:39:46,467][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:39:46,790][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:47,113][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:47,437][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:47,763][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:48,085][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:48,409][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:48,730][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:49,053][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:49,376][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:49,698][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:50,020][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:50,343][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:50,670][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:50,992][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:51,315][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:51,643][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:52,349][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:53,082][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:53,083][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:53,086][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:54,101][__main__][INFO] - Iteration 548 took 23s (40.63% Gen, 55.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 11m 56s. Estimated total time: 19h 45m 1s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 30s.
[2025-11-13 11:39:54,103][__main__][INFO] - Starting iteration 548.
[2025-11-13 11:39:54,106][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:39:54,107][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:03,472][__main__][INFO] - Number of regex retries in iteration 548: 0
[2025-11-13 11:40:03,472][__main__][INFO] - agents played in iteration 548 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:40:03,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:03,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,038][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:04,038][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:04,764][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:05,059][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:05,383][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:05,706][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:06,030][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:06,353][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:06,677][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:07,002][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:07,328][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:07,655][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:07,977][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:08,303][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:08,625][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:08,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:09,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:09,594][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:09,917][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:10,241][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:10,563][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:10,886][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:11,209][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:11,534][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:11,858][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:12,180][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:12,503][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:40:12,825][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:40:13,147][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:40:13,469][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:40:13,792][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:40:14,115][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:40:14,439][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:40:14,762][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:40:15,087][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:40:15,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:40:16,489][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:40:16,491][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:40:16,492][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:40:17,455][__main__][INFO] - Iteration 549 took 23s (40.11% Gen, 55.76% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 1s. Estimated total time: 19h 27m 29s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 54s, 500 more iterations: 3h 14m 34s.
[2025-11-13 11:40:17,457][__main__][INFO] - Starting iteration 549.
[2025-11-13 11:40:17,461][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:40:17,462][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:26,775][__main__][INFO] - Number of regex retries in iteration 549: 0
[2025-11-13 11:40:26,776][__main__][INFO] - agents played in iteration 549 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:40:27,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,272][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,305][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,339][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:27,339][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:28,069][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:28,363][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:28,687][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:29,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:29,333][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:29,658][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:29,982][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:30,308][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:30,630][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:30,953][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:31,278][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:31,602][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:31,926][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:32,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:32,580][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:32,903][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:33,227][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:33,875][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:34,200][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:34,527][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:34,851][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:35,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:35,497][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:35,821][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:40:36,146][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:40:36,471][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:40:36,796][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:40:37,124][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:40:37,446][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:40:37,771][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:40:38,095][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:40:38,418][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:40:39,120][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:40:39,861][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:40:39,862][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:40:39,864][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:40:40,736][__main__][INFO] - Iteration 550 took 23s (40.02% Gen, 56.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 49m 57s. Estimated total time: 19h 23m 48s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 58s.
[2025-11-13 11:40:40,739][__main__][INFO] - Starting iteration 550.
[2025-11-13 11:40:40,742][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:40:40,743][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:50,052][__main__][INFO] - Number of regex retries in iteration 550: 0
[2025-11-13 11:40:50,053][__main__][INFO] - agents played in iteration 550 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:40:50,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:50,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:50,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:50,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:50,608][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:50,609][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:51,328][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:51,623][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:51,947][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:52,270][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:52,593][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:52,916][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:53,244][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:53,572][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:53,901][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:54,229][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:54,557][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:54,883][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:55,207][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:55,536][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:55,862][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:56,185][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:56,511][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:56,836][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:57,162][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:57,485][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:57,811][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:58,135][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:58,459][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:58,781][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:59,106][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:40:59,430][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:40:59,753][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:00,075][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:00,399][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:00,720][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:01,045][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:01,371][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:01,698][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:02,352][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:03,075][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:03,077][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:03,079][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:05,078][__main__][INFO] - Iteration 551 took 24s (38.25% Gen, 53.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 42m 34s. Estimated total time: 20h 16m 50s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 33s, 500 more iterations: 3h 22m 48s.
[2025-11-13 11:41:05,080][__main__][INFO] - Starting iteration 551.
[2025-11-13 11:41:05,083][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:05,083][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:41:14,519][__main__][INFO] - Number of regex retries in iteration 551: 0
[2025-11-13 11:41:14,520][__main__][INFO] - agents played in iteration 551 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:41:14,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,014][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,049][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,083][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,083][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:41:15,083][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:41:15,806][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:41:16,101][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:41:16,426][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:41:16,750][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:41:17,074][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:41:17,398][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:41:17,721][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:41:18,044][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:41:18,368][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:41:18,691][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:41:19,014][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:41:19,336][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:41:19,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:41:19,982][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:41:20,304][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:41:20,627][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:41:20,951][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:41:21,275][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:41:21,599][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:41:21,921][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:41:22,245][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:41:22,567][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:41:22,889][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:41:23,212][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:41:23,535][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:41:23,858][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:41:24,182][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:24,506][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:24,832][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:25,154][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:25,478][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:25,802][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:26,125][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:26,804][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:27,534][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:27,535][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:27,538][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:28,535][__main__][INFO] - Iteration 552 took 23s (40.24% Gen, 55.50% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 58m 1s. Estimated total time: 19h 32m 40s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 5s, 500 more iterations: 3h 15m 26s.
[2025-11-13 11:41:28,537][__main__][INFO] - Starting iteration 552.
[2025-11-13 11:41:28,541][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:28,542][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:41:37,606][__main__][INFO] - Number of regex retries in iteration 552: 0
[2025-11-13 11:41:37,607][__main__][INFO] - agents played in iteration 552 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:41:38,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:41:38,177][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:41:38,904][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:41:39,200][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:41:39,528][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:41:39,855][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:41:40,181][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:41:40,505][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:41:40,831][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:41:41,159][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:41:41,484][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:41:41,807][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:41:42,131][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:41:42,453][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:41:42,777][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:41:43,102][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:41:43,426][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:41:43,751][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:41:44,076][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:41:44,400][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:41:44,723][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:41:45,044][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:41:45,366][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:41:45,689][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:41:46,011][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:41:46,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:41:46,658][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:41:46,980][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:41:47,302][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:47,624][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:47,947][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:48,270][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:48,592][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:48,915][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:49,238][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:49,903][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:50,647][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:50,648][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:50,650][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:51,701][__main__][INFO] - Iteration 553 took 23s (39.14% Gen, 56.31% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 43m 1s. Estimated total time: 19h 18m 3s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 0s.
[2025-11-13 11:41:51,703][__main__][INFO] - Starting iteration 553.
[2025-11-13 11:41:51,707][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:51,707][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:01,147][__main__][INFO] - Number of regex retries in iteration 553: 0
[2025-11-13 11:42:01,147][__main__][INFO] - agents played in iteration 553 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:42:01,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:01,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:01,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:01,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:01,713][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:01,713][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:02,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:02,737][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:03,063][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:03,386][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:03,709][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:04,032][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:04,355][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:04,678][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:05,002][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:05,325][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:05,649][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:05,972][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:06,296][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:06,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:06,948][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:07,274][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:07,598][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:07,922][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:08,246][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:08,569][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:08,894][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:09,217][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:09,547][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:09,869][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:10,191][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:10,514][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:10,842][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:11,166][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:11,494][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:11,819][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:12,142][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:42:12,463][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:42:12,786][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:42:13,449][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:42:14,190][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:42:14,191][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:42:14,194][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:42:15,321][__main__][INFO] - Iteration 554 took 23s (39.97% Gen, 55.25% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 5m 19s. Estimated total time: 19h 40m 45s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 47s.
[2025-11-13 11:42:15,323][__main__][INFO] - Starting iteration 554.
[2025-11-13 11:42:15,327][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:42:15,327][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:24,656][__main__][INFO] - Number of regex retries in iteration 554: 0
[2025-11-13 11:42:24,656][__main__][INFO] - agents played in iteration 554 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:42:25,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,209][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:25,209][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:25,931][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:26,226][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:26,550][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:26,873][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:27,201][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:27,523][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:27,846][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:28,169][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:28,492][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:28,816][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:29,140][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:29,462][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:29,785][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:30,108][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:30,432][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:30,755][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:31,078][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:31,402][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:31,724][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:32,046][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:32,368][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:32,691][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:33,012][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:33,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:33,660][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:33,982][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:34,304][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:34,626][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:34,947][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:35,270][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:35,591][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:42:35,913][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:42:36,236][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:42:36,899][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:42:37,630][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:42:37,631][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:42:37,633][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:42:38,722][__main__][INFO] - Iteration 555 took 23s (39.87% Gen, 55.47% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 53m 58s. Estimated total time: 19h 29m 48s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 59s, 500 more iterations: 3h 14m 58s.
[2025-11-13 11:42:38,724][__main__][INFO] - Starting iteration 555.
[2025-11-13 11:42:38,727][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:42:38,728][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:48,431][__main__][INFO] - Number of regex retries in iteration 555: 0
[2025-11-13 11:42:48,431][__main__][INFO] - agents played in iteration 555 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:42:48,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:48,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:48,953][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:48,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:48,988][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:48,988][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:49,715][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:50,010][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:50,335][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:50,657][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:50,986][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:51,308][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:51,631][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:51,954][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:52,277][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:52,600][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:52,923][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:53,245][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:53,570][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:53,894][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:54,220][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:54,544][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:54,868][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:55,192][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:55,517][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:55,845][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:56,175][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:56,500][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:56,824][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:57,147][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:57,471][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:57,794][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:58,121][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:58,446][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:58,771][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:59,100][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:59,429][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:42:59,754][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:00,078][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:00,764][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:01,494][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:01,496][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:01,497][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:02,614][__main__][INFO] - Iteration 556 took 23s (40.62% Gen, 54.70% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 18m 9s. Estimated total time: 19h 54m 23s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 48s, 500 more iterations: 3h 19m 3s.
[2025-11-13 11:43:02,616][__main__][INFO] - Starting iteration 556.
[2025-11-13 11:43:02,619][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:43:02,619][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:43:11,975][__main__][INFO] - Number of regex retries in iteration 556: 0
[2025-11-13 11:43:11,975][__main__][INFO] - agents played in iteration 556 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:43:12,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:12,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:12,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:12,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:12,539][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:43:12,539][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:43:13,273][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:43:13,568][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:43:13,892][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:43:14,216][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:43:14,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:43:14,863][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:43:15,189][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:43:15,515][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:43:15,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:43:16,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:43:16,487][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:43:16,812][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:43:17,142][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:43:17,464][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:43:17,790][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:43:18,115][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:43:18,438][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:43:18,762][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:43:19,084][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:43:19,408][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:43:19,732][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:43:20,057][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:43:20,380][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:43:20,704][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:43:21,029][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:43:21,356][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:43:21,680][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:43:22,004][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:43:22,327][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:43:22,650][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:43:22,971][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:43:23,297][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:23,620][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:24,289][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:25,038][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:25,039][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:25,041][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:25,986][__main__][INFO] - Iteration 557 took 23s (40.04% Gen, 55.91% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 51m 48s. Estimated total time: 19h 28m 25s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 44s.
[2025-11-13 11:43:25,988][__main__][INFO] - Starting iteration 557.
[2025-11-13 11:43:25,992][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:43:25,992][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:43:35,944][__main__][INFO] - Number of regex retries in iteration 557: 0
[2025-11-13 11:43:35,945][__main__][INFO] - agents played in iteration 557 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:43:36,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:36,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:36,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:36,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:36,502][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:43:36,502][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:43:37,227][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:43:37,522][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:43:37,848][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:43:38,174][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:43:38,497][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:43:38,820][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:43:39,145][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:43:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:43:39,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:43:40,119][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:43:40,442][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:43:40,764][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:43:41,088][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:43:41,417][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:43:41,739][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:43:42,064][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:43:42,389][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:43:42,714][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:43:43,040][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:43:43,363][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:43:43,688][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:43:44,012][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:43:44,335][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:43:44,658][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:43:44,981][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:43:45,309][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:43:45,633][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:43:45,957][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:43:46,281][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:43:46,604][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:43:46,926][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:43:47,253][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:47,578][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:48,252][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:48,988][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:48,989][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:48,991][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:50,000][__main__][INFO] - Iteration 558 took 24s (41.45% Gen, 54.34% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 23m 25s. Estimated total time: 20h 0m 26s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 4s.
[2025-11-13 11:43:50,002][__main__][INFO] - Starting iteration 558.
[2025-11-13 11:43:50,005][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:43:50,006][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:43:59,768][__main__][INFO] - Number of regex retries in iteration 558: 0
[2025-11-13 11:43:59,769][__main__][INFO] - agents played in iteration 558 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:44:00,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:00,254][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:00,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:00,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:00,320][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:00,321][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:01,035][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:01,329][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:01,653][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:01,976][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:02,300][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:02,627][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:02,951][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:03,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:03,597][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:03,921][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:04,246][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:04,568][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:04,892][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:05,215][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:05,539][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:05,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:06,186][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:06,508][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:06,830][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:07,153][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:07,476][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:07,799][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:08,121][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:08,444][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:08,766][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:09,089][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:09,413][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:09,735][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:10,058][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:10,381][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:10,704][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:11,028][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:44:11,352][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:44:12,018][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 11:44:12,754][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:44:12,756][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:44:12,757][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:44:13,762][__main__][INFO] - Iteration 559 took 23s (41.10% Gen, 54.67% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 10m 28s. Estimated total time: 19h 47m 52s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 58s.
[2025-11-13 11:44:13,764][__main__][INFO] - Starting iteration 559.
[2025-11-13 11:44:13,767][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:44:13,768][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:44:23,105][__main__][INFO] - Number of regex retries in iteration 559: 0
[2025-11-13 11:44:23,106][__main__][INFO] - agents played in iteration 559 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:44:23,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:23,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:23,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:23,654][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:23,655][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:23,656][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:24,359][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:24,656][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:24,980][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:25,304][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:25,633][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:25,958][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:26,282][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:26,607][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:26,932][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:27,258][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:27,583][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:27,906][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:28,235][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:28,561][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:28,884][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:29,209][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:29,534][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:29,858][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:30,182][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:30,504][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:30,826][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:31,149][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:31,470][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:31,793][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:32,117][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:32,440][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:32,765][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:33,087][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:33,410][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:33,732][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:34,057][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:34,382][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:44:34,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:44:35,370][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:44:36,107][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:44:36,109][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:44:36,111][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:44:36,957][__main__][INFO] - Iteration 560 took 23s (40.26% Gen, 56.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 41m 45s. Estimated total time: 19h 19m 33s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 39s, 500 more iterations: 3h 13m 15s.
[2025-11-13 11:44:36,959][__main__][INFO] - Starting iteration 560.
[2025-11-13 11:44:36,963][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:44:36,963][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:44:46,115][__main__][INFO] - Number of regex retries in iteration 560: 0
[2025-11-13 11:44:46,116][__main__][INFO] - agents played in iteration 560 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:44:46,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:46,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:46,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:46,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:46,675][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:46,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:47,412][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:47,707][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:48,031][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:48,353][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:48,678][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:49,002][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:49,326][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:49,652][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:49,981][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:50,305][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:50,631][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:50,954][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:51,277][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:51,602][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:51,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:52,250][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:52,575][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:52,901][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:53,224][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:53,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:53,877][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:54,201][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:54,523][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:54,846][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:55,168][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:55,492][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:55,814][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:56,137][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:56,460][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:56,782][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:57,105][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:57,428][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:44:57,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:44:58,411][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:44:59,143][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:44:59,145][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:44:59,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:01,099][__main__][INFO] - Iteration 561 took 24s (37.92% Gen, 53.99% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 28m 38s. Estimated total time: 20h 6m 50s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 13s, 500 more iterations: 3h 21m 8s.
[2025-11-13 11:45:01,101][__main__][INFO] - Starting iteration 561.
[2025-11-13 11:45:01,104][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:01,104][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:11,066][__main__][INFO] - Number of regex retries in iteration 561: 0
[2025-11-13 11:45:11,067][__main__][INFO] - agents played in iteration 561 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:45:11,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:11,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:11,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:11,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:11,623][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:11,623][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:12,356][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:45:12,650][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:45:12,975][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:45:13,298][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:45:13,625][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:45:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:45:14,273][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:45:14,596][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:45:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:45:15,246][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:45:15,575][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:45:15,897][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:45:16,226][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:45:16,550][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:45:16,874][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:45:17,200][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:45:17,525][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:45:17,848][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:45:18,170][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:45:18,492][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:45:18,816][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:45:19,138][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:45:19,461][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:45:19,784][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:45:20,105][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:45:20,428][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:45:20,751][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:45:21,073][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:45:21,397][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:45:21,720][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:45:22,043][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:45:22,365][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:45:22,693][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:45:23,375][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:45:24,126][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:45:24,127][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:45:24,129][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:25,141][__main__][INFO] - Iteration 562 took 24s (41.45% Gen, 54.34% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 23m 18s. Estimated total time: 20h 1m 54s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 19s.
[2025-11-13 11:45:25,143][__main__][INFO] - Starting iteration 562.
[2025-11-13 11:45:25,146][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:25,146][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:34,741][__main__][INFO] - Number of regex retries in iteration 562: 0
[2025-11-13 11:45:34,742][__main__][INFO] - agents played in iteration 562 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:45:35,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:35,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:35,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:35,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:35,297][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:35,297][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:36,022][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:45:36,318][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:45:36,642][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:45:36,969][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:45:37,293][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:45:37,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:45:37,947][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:45:38,272][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:45:38,598][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:45:38,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:45:39,249][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:45:39,576][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:45:39,899][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:45:40,223][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:45:40,550][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:45:40,875][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:45:41,199][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:45:41,523][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:45:41,848][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:45:42,173][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:45:42,498][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:45:42,821][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:45:43,145][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:45:43,468][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:45:43,790][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:45:44,113][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:45:44,435][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:45:44,757][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:45:45,080][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:45:45,403][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:45:45,725][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:45:46,050][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:45:46,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:45:47,032][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:45:47,758][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:45:47,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:45:47,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:48,825][__main__][INFO] - Iteration 563 took 23s (40.52% Gen, 54.98% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 4m 59s. Estimated total time: 19h 43m 59s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 27s, 500 more iterations: 3h 17m 19s.
[2025-11-13 11:45:48,827][__main__][INFO] - Starting iteration 563.
[2025-11-13 11:45:48,830][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:48,830][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:58,189][__main__][INFO] - Number of regex retries in iteration 563: 0
[2025-11-13 11:45:58,190][__main__][INFO] - agents played in iteration 563 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:45:58,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:58,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:58,715][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:58,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:58,750][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:58,750][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:59,486][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:45:59,781][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:00,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:00,431][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:00,756][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:01,080][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:01,403][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:01,727][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:02,052][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:02,380][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:02,706][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:03,029][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:03,354][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:03,677][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:04,001][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:04,325][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:04,651][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:04,976][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:05,303][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:05,626][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:05,949][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:06,277][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:06,607][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:06,933][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:07,256][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:07,584][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:07,905][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:08,229][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:08,552][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:08,875][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:09,198][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:09,521][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:09,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:10,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:11,276][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:11,278][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:11,279][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:12,316][__main__][INFO] - Iteration 564 took 23s (39.85% Gen, 55.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 56s. Estimated total time: 19h 34m 20s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 8s, 500 more iterations: 3h 15m 43s.
[2025-11-13 11:46:12,318][__main__][INFO] - Starting iteration 564.
[2025-11-13 11:46:12,321][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:12,322][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:46:21,779][__main__][INFO] - Number of regex retries in iteration 564: 0
[2025-11-13 11:46:21,779][__main__][INFO] - agents played in iteration 564 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:46:22,233][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,300][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,333][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,334][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:46:22,335][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:46:23,070][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:46:23,364][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:23,689][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:24,012][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:24,338][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:24,668][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:24,996][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:25,320][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:25,645][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:25,968][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:26,290][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:26,613][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:26,936][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:27,260][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:27,584][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:27,907][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:28,229][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:28,553][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:28,876][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:29,199][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:29,522][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:29,847][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:30,172][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:30,493][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:30,815][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:31,138][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:31,461][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:31,783][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:32,107][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:32,429][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:32,752][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:33,075][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:33,399][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:34,100][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:34,831][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:34,833][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:34,834][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:35,837][__main__][INFO] - Iteration 565 took 23s (40.22% Gen, 55.51% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 56m 3s. Estimated total time: 19h 35m 50s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 58s.
[2025-11-13 11:46:35,839][__main__][INFO] - Starting iteration 565.
[2025-11-13 11:46:35,843][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:35,843][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:46:45,167][__main__][INFO] - Number of regex retries in iteration 565: 0
[2025-11-13 11:46:45,168][__main__][INFO] - agents played in iteration 565 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:46:45,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:45,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:45,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:45,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:45,723][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:46:45,723][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:46:46,443][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:46:46,738][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:47,062][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:47,385][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:47,713][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:48,038][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:48,363][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:48,691][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:49,015][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:49,344][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:49,669][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:49,995][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:50,320][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:50,643][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:50,967][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:51,292][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:51,617][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:51,944][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:52,270][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:52,596][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:52,921][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:53,246][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:53,573][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:53,899][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:54,544][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:54,868][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:55,191][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:55,515][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:55,844][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:56,169][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:56,492][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:56,816][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:57,477][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:58,190][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:58,191][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:58,193][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:59,237][__main__][INFO] - Iteration 566 took 23s (39.85% Gen, 55.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 49m 36s. Estimated total time: 19h 29m 46s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 59s, 500 more iterations: 3h 14m 57s.
[2025-11-13 11:46:59,239][__main__][INFO] - Starting iteration 566.
[2025-11-13 11:46:59,243][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:59,244][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:08,405][__main__][INFO] - Number of regex retries in iteration 566: 0
[2025-11-13 11:47:08,405][__main__][INFO] - agents played in iteration 566 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:47:08,859][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:08,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:08,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:08,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:08,960][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:08,961][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:09,707][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:10,002][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:10,326][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:10,649][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:10,972][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:11,295][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:11,618][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:11,946][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:47:12,270][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:47:12,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:47:12,917][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:47:13,241][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:47:13,564][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:47:13,887][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:47:14,210][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:47:14,534][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:47:14,859][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:47:15,183][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:47:15,506][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:47:15,831][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:47:16,155][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:47:16,478][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:47:16,800][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:47:17,123][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:47:17,449][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:47:17,770][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:47:18,093][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:47:18,418][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:47:18,741][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:47:19,069][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:47:19,391][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:47:19,713][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:47:20,036][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:47:20,703][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:47:21,420][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:47:21,422][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:47:21,423][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:47:22,471][__main__][INFO] - Iteration 567 took 23s (39.44% Gen, 56.05% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 40m 52s. Estimated total time: 19h 21m 26s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 34s.
[2025-11-13 11:47:22,473][__main__][INFO] - Starting iteration 567.
[2025-11-13 11:47:22,476][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:47:22,477][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:31,569][__main__][INFO] - Number of regex retries in iteration 567: 0
[2025-11-13 11:47:31,570][__main__][INFO] - agents played in iteration 567 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:47:32,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,089][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,123][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:32,124][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:32,849][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:33,145][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:33,469][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:33,792][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:34,116][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:34,440][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:34,768][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:35,095][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:47:35,418][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:47:35,746][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:47:36,068][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:47:36,392][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:47:36,718][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:47:37,045][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:47:37,371][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:47:37,694][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:47:38,017][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:47:38,341][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:47:38,666][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:47:38,990][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:47:39,313][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:47:39,635][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:47:39,958][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:47:40,280][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:47:40,602][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:47:40,924][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:47:41,251][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:47:41,579][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:47:41,902][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:47:42,224][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:47:42,548][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:47:42,870][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:47:43,193][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:47:43,859][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:47:44,579][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:47:44,580][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:47:44,582][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:47:45,594][__main__][INFO] - Iteration 568 took 23s (39.33% Gen, 56.29% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 34m 59s. Estimated total time: 19h 15m 55s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 31s, 500 more iterations: 3h 12m 39s.
[2025-11-13 11:47:45,596][__main__][INFO] - Starting iteration 568.
[2025-11-13 11:47:45,600][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:47:45,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:55,041][__main__][INFO] - Number of regex retries in iteration 568: 0
[2025-11-13 11:47:55,043][__main__][INFO] - agents played in iteration 568 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:47:55,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:55,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:55,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:55,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:55,593][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:55,594][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:56,312][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:56,606][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:56,931][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:57,254][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:57,577][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:57,902][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:58,224][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:58,549][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:47:58,873][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:47:59,196][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:47:59,521][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:47:59,844][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:00,167][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:00,491][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:00,815][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:01,139][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:01,463][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:01,788][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:02,114][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:02,439][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:02,762][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:03,085][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:03,407][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:03,729][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:04,051][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:04,374][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:04,699][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:05,022][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:05,345][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:05,667][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:05,991][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:06,314][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:06,639][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:07,328][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:08,061][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:08,063][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:08,065][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:09,083][__main__][INFO] - Iteration 569 took 23s (40.21% Gen, 55.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 52m 52s. Estimated total time: 19h 34m 13s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 8s, 500 more iterations: 3h 15m 42s.
[2025-11-13 11:48:09,085][__main__][INFO] - Starting iteration 569.
[2025-11-13 11:48:09,088][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:48:09,089][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:48:18,469][__main__][INFO] - Number of regex retries in iteration 569: 0
[2025-11-13 11:48:18,469][__main__][INFO] - agents played in iteration 569 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:48:18,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:18,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:18,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:19,020][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:19,021][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:48:19,022][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:48:19,770][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:48:20,065][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:48:20,391][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:48:20,714][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:48:21,041][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:48:21,366][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:48:21,693][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:48:22,019][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:48:22,347][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:48:22,677][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:48:23,002][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:48:23,328][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:23,655][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:23,980][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:24,304][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:24,627][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:24,952][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:25,275][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:25,601][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:25,923][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:26,246][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:26,569][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:26,891][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:27,213][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:27,537][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:27,859][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:28,182][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:28,505][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:28,831][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:29,153][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:29,479][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:29,801][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:30,127][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:30,794][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:31,517][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:31,518][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:31,520][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:32,542][__main__][INFO] - Iteration 570 took 23s (39.99% Gen, 55.65% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 50m 59s. Estimated total time: 19h 32m 43s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 5s, 500 more iterations: 3h 15m 27s.
[2025-11-13 11:48:32,544][__main__][INFO] - Starting iteration 570.
[2025-11-13 11:48:32,547][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:48:32,548][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:48:41,950][__main__][INFO] - Number of regex retries in iteration 570: 0
[2025-11-13 11:48:41,951][__main__][INFO] - agents played in iteration 570 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:48:42,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:42,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:42,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:42,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:42,501][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:48:42,501][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:48:43,215][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:48:43,510][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:48:43,834][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:48:44,157][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:48:44,481][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:48:44,804][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:48:45,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:48:45,452][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:48:45,778][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:48:46,102][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:48:46,426][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:48:46,749][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:47,073][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:47,395][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:47,722][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:48,051][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:48,377][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:48,704][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:49,029][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:49,353][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:49,676][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:50,004][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:50,332][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:50,657][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:50,980][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:51,304][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:51,628][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:51,951][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:52,273][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:52,596][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:52,919][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:53,240][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:53,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:54,219][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:54,931][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:54,932][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:54,934][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:57,046][__main__][INFO] - Iteration 571 took 24s (38.38% Gen, 53.00% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 42m 50s. Estimated total time: 20h 24m 58s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 49s, 500 more iterations: 3h 24m 9s.
[2025-11-13 11:48:57,048][__main__][INFO] - Starting iteration 571.
[2025-11-13 11:48:57,051][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:48:57,051][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:07,531][__main__][INFO] - Number of regex retries in iteration 571: 0
[2025-11-13 11:49:07,532][__main__][INFO] - agents played in iteration 571 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:49:08,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,102][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:08,102][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:08,826][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:09,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:09,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:09,767][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:10,091][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:10,417][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:10,739][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:11,064][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:49:11,386][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:49:11,709][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:49:12,031][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:49:12,360][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:49:12,684][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:49:13,009][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:49:13,337][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:49:13,660][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:49:13,983][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:49:14,308][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:49:14,631][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:49:14,955][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:49:15,279][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:49:15,601][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:49:15,925][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:49:16,248][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:49:16,570][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:49:16,892][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:49:17,214][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:49:17,536][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:49:17,862][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:49:18,184][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:49:18,507][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:49:18,830][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:49:19,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:49:19,821][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:49:20,560][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:49:20,562][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:49:20,563][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:49:21,636][__main__][INFO] - Iteration 572 took 24s (42.63% Gen, 53.01% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 46m 45s. Estimated total time: 20h 29m 17s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 58s, 500 more iterations: 3h 24m 52s.
[2025-11-13 11:49:21,638][__main__][INFO] - Starting iteration 572.
[2025-11-13 11:49:21,641][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:49:21,642][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:31,798][__main__][INFO] - Number of regex retries in iteration 572: 0
[2025-11-13 11:49:31,799][__main__][INFO] - agents played in iteration 572 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:49:32,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:32,285][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:32,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:32,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:32,351][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:32,351][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:33,056][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:33,350][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:33,675][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:34,000][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:34,323][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:34,646][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:34,968][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:35,290][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:49:35,612][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:49:35,936][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:49:36,260][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:49:36,588][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:49:36,910][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:49:37,232][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:49:37,555][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:49:37,877][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:49:38,199][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:49:38,520][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:49:38,843][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:49:39,165][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:49:39,489][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:49:39,810][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:49:40,132][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:49:40,455][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:49:40,777][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:49:41,100][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:49:41,422][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:49:41,743][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:49:42,066][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:49:42,387][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:49:42,708][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:49:43,030][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:49:43,352][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:49:44,050][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:49:44,795][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:49:44,796][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:49:44,798][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:49:45,798][__main__][INFO] - Iteration 573 took 24s (42.04% Gen, 53.81% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 24m 57s. Estimated total time: 20h 7m 54s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 15s, 500 more iterations: 3h 21m 19s.
[2025-11-13 11:49:45,800][__main__][INFO] - Starting iteration 573.
[2025-11-13 11:49:45,803][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:49:45,804][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:56,320][__main__][INFO] - Number of regex retries in iteration 573: 0
[2025-11-13 11:49:56,321][__main__][INFO] - agents played in iteration 573 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:49:56,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:56,821][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:56,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:56,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:56,888][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:56,888][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:57,590][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:57,886][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:58,212][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:58,533][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:58,856][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:59,179][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:59,501][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:59,825][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:50:00,150][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:50:00,473][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:50:00,794][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:50:01,119][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:50:01,442][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:50:01,763][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:02,087][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:02,410][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:02,733][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:03,055][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:03,378][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:04,023][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:04,345][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:04,667][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:04,989][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:05,311][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:05,633][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:05,957][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:06,279][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:06,601][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:06,926][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:07,250][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:07,578][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:07,906][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:08,608][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:09,310][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:09,312][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:09,313][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:10,274][__main__][INFO] - Iteration 574 took 24s (42.97% Gen, 53.09% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 40m 16s. Estimated total time: 20h 23m 37s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 47s, 500 more iterations: 3h 23m 56s.
[2025-11-13 11:50:10,276][__main__][INFO] - Starting iteration 574.
[2025-11-13 11:50:10,279][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:10,280][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:50:19,962][__main__][INFO] - Number of regex retries in iteration 574: 0
[2025-11-13 11:50:19,962][__main__][INFO] - agents played in iteration 574 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:50:20,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:20,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:20,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:20,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:20,847][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:50:20,847][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:50:21,582][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:50:21,878][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:50:22,202][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:50:22,523][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:50:22,847][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:50:23,175][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:50:23,500][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:50:23,827][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:50:24,153][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:50:24,478][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:50:24,806][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:50:25,134][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:50:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:50:25,788][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:26,112][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:26,436][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:26,764][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:27,088][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:27,411][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:27,734][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:28,055][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:28,377][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:28,701][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:29,025][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:29,349][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:29,673][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:29,996][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:30,319][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:30,644][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:30,972][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:31,297][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:31,620][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:31,947][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:32,679][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:33,377][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:33,378][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:33,380][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:34,353][__main__][INFO] - Iteration 575 took 24s (40.22% Gen, 55.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 19m 59s. Estimated total time: 20h 3m 44s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 7s, 500 more iterations: 3h 20m 37s.
[2025-11-13 11:50:34,355][__main__][INFO] - Starting iteration 575.
[2025-11-13 11:50:34,358][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:34,358][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:50:43,959][__main__][INFO] - Number of regex retries in iteration 575: 0
[2025-11-13 11:50:43,960][__main__][INFO] - agents played in iteration 575 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:50:44,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:44,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:44,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:44,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:44,521][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:50:44,521][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:50:45,224][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:50:45,518][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:50:45,841][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:50:46,164][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:50:46,489][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:50:46,814][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:50:47,140][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:50:47,467][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:50:47,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:50:48,119][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:50:48,445][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:50:48,772][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:50:49,097][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:50:49,422][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:49,747][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:50,070][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:50,391][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:50,716][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:51,039][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:51,362][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:51,684][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:52,006][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:52,328][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:52,650][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:52,974][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:53,298][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:53,620][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:53,942][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:54,265][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:54,588][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:54,911][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:55,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:56,268][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:56,968][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:56,970][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:56,971][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:57,917][__main__][INFO] - Iteration 576 took 23s (40.75% Gen, 55.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 53m 52s. Estimated total time: 19h 38m 1s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 20s.
[2025-11-13 11:50:57,919][__main__][INFO] - Starting iteration 576.
[2025-11-13 11:50:57,922][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:57,923][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:07,949][__main__][INFO] - Number of regex retries in iteration 576: 0
[2025-11-13 11:51:07,950][__main__][INFO] - agents played in iteration 576 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:51:08,394][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:08,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:08,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:08,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:08,495][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:08,495][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:09,180][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:09,474][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:09,798][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:10,123][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:10,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:10,768][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:11,091][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:11,414][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:11,738][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:51:12,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:51:12,387][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:51:12,709][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:51:13,031][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:51:13,355][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:51:13,677][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:51:13,999][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:51:14,323][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:51:14,645][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:51:14,968][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:51:15,290][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:51:15,614][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:51:15,934][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:51:16,256][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:51:16,579][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:51:16,902][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:51:17,225][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:51:17,547][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:51:17,869][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:51:18,193][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:51:18,515][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:51:18,838][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:51:19,163][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:51:19,487][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:51:20,194][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:51:20,888][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:51:20,889][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:51:20,893][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:51:21,768][__main__][INFO] - Iteration 577 took 23s (42.05% Gen, 54.28% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 7m 48s. Estimated total time: 19h 52m 20s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 44s, 500 more iterations: 3h 18m 43s.
[2025-11-13 11:51:21,770][__main__][INFO] - Starting iteration 577.
[2025-11-13 11:51:21,773][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:51:21,774][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:31,532][__main__][INFO] - Number of regex retries in iteration 577: 0
[2025-11-13 11:51:31,533][__main__][INFO] - agents played in iteration 577 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:51:31,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:32,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:32,052][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:32,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:32,086][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:32,087][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:32,780][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:33,073][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:33,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:33,721][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:34,044][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:34,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:34,690][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:35,012][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:35,335][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:51:35,658][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:51:35,983][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:51:36,306][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:51:36,629][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:51:36,954][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:51:37,282][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:51:37,611][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:51:37,936][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:51:38,259][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:51:38,585][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:51:38,908][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:51:39,230][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:51:39,552][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:51:39,877][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:51:40,201][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:51:40,523][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:51:40,846][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:51:41,168][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:51:41,491][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:51:41,813][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:51:42,135][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:51:42,459][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:51:42,782][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:51:43,105][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:51:43,816][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:51:44,538][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:51:44,540][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:51:44,542][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:51:45,618][__main__][INFO] - Iteration 578 took 23s (40.93% Gen, 54.55% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 21s. Estimated total time: 19h 52m 18s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 44s, 500 more iterations: 3h 18m 43s.
[2025-11-13 11:51:45,620][__main__][INFO] - Starting iteration 578.
[2025-11-13 11:51:45,624][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:51:45,625][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:55,953][__main__][INFO] - Number of regex retries in iteration 578: 0
[2025-11-13 11:51:55,954][__main__][INFO] - agents played in iteration 578 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:51:56,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,480][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,514][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:56,515][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:57,228][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:57,523][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:57,849][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:58,173][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:58,496][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:58,819][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:59,142][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:59,465][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:59,789][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:52:00,112][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:00,434][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:00,756][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:01,079][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:01,402][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:01,725][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:02,047][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:02,368][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:02,693][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:03,021][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:03,344][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:03,669][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:03,992][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:04,321][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:04,645][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:04,967][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:05,289][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:05,611][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:05,937][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:06,262][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:06,585][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:06,909][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:07,555][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:08,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:08,977][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:08,979][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:08,981][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:10,046][__main__][INFO] - Iteration 579 took 24s (42.29% Gen, 53.34% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 35m 51s. Estimated total time: 20h 21m 12s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 32s.
[2025-11-13 11:52:10,049][__main__][INFO] - Starting iteration 579.
[2025-11-13 11:52:10,052][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:52:10,053][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:52:20,285][__main__][INFO] - Number of regex retries in iteration 579: 0
[2025-11-13 11:52:20,286][__main__][INFO] - agents played in iteration 579 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:52:20,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,845][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:52:20,845][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:52:21,546][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:52:21,842][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:52:22,165][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:52:22,487][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:52:22,810][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:52:23,135][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:52:23,458][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:52:23,783][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:52:24,105][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:52:24,430][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:24,752][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:25,074][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:25,396][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:25,721][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:26,043][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:26,367][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:26,691][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:27,014][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:27,338][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:27,660][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:27,984][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:28,306][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:28,629][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:28,949][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:29,271][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:29,594][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:29,917][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:30,240][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:30,563][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:30,887][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:31,209][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:31,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:31,855][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:32,563][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:33,285][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:33,286][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:33,288][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:34,270][__main__][INFO] - Iteration 580 took 24s (42.25% Gen, 53.69% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 25m 9s. Estimated total time: 20h 10m 55s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 21s, 500 more iterations: 3h 21m 49s.
[2025-11-13 11:52:34,272][__main__][INFO] - Starting iteration 580.
[2025-11-13 11:52:34,274][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:52:34,275][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:52:44,491][__main__][INFO] - Number of regex retries in iteration 580: 0
[2025-11-13 11:52:44,492][__main__][INFO] - agents played in iteration 580 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:52:44,939][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:44,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:45,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:45,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:45,041][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:52:45,042][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:52:45,748][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:52:46,042][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:52:46,370][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:52:46,700][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:52:47,029][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:52:47,358][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:52:47,683][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:52:48,009][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:52:48,338][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:52:48,661][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:48,983][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:49,306][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:49,629][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:49,951][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:50,273][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:50,598][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:50,922][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:51,244][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:51,567][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:51,888][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:52,212][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:52,535][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:52,858][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:53,181][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:53,507][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:53,836][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:54,162][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:54,487][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:54,809][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:55,138][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:55,461][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:55,784][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:56,108][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:56,822][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:57,547][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:57,549][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:57,551][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:59,363][__main__][INFO] - Iteration 581 took 25s (40.72% Gen, 52.05% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 8m 17s. Estimated total time: 20h 54m 27s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 48s, 500 more iterations: 3h 29m 4s.
[2025-11-13 11:52:59,365][__main__][INFO] - Starting iteration 581.
[2025-11-13 11:52:59,367][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:52:59,368][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:10,090][__main__][INFO] - Number of regex retries in iteration 581: 0
[2025-11-13 11:53:10,091][__main__][INFO] - agents played in iteration 581 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:53:10,530][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:10,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:10,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:10,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:10,629][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:10,629][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:11,311][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:53:11,607][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:53:11,933][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:53:12,258][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:53:12,585][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:53:12,907][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:53:13,230][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:53:13,553][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:53:13,877][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:53:14,199][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:53:14,524][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:53:14,846][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:53:15,168][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:53:15,491][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:53:15,813][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:53:16,136][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:53:16,459][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:53:16,782][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:53:17,104][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:53:17,427][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:53:17,750][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:53:18,073][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:53:18,394][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:53:18,717][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:53:19,040][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:53:19,365][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:53:19,691][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:53:20,014][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:53:20,339][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:53:20,663][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:53:20,987][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:53:21,315][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:53:21,642][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:53:22,358][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:53:23,049][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:53:23,050][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:53:23,052][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:53:23,992][__main__][INFO] - Iteration 582 took 24s (43.54% Gen, 52.64% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 44m 41s. Estimated total time: 20h 31m 16s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 2s, 500 more iterations: 3h 25m 12s.
[2025-11-13 11:53:23,994][__main__][INFO] - Starting iteration 582.
[2025-11-13 11:53:23,997][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:53:23,997][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:34,583][__main__][INFO] - Number of regex retries in iteration 582: 0
[2025-11-13 11:53:34,583][__main__][INFO] - agents played in iteration 582 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:53:35,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:35,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:35,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:35,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:35,131][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:35,132][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:35,840][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:53:36,134][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:53:36,458][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:53:36,782][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:53:37,106][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:53:37,431][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:53:37,755][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:53:38,080][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:53:38,404][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:53:38,727][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:53:39,050][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:53:39,372][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:53:39,694][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:53:40,017][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:53:40,340][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:53:40,663][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:53:40,986][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:53:41,309][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:53:41,631][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:53:41,953][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:53:42,276][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:53:42,601][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:53:42,924][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:53:43,249][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:53:43,574][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:53:43,903][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:53:44,230][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:53:44,556][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:53:44,879][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:53:45,204][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:53:45,528][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:53:45,854][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:53:46,178][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:53:46,906][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:53:47,615][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:53:47,618][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:53:47,620][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:53:48,621][__main__][INFO] - Iteration 583 took 24s (42.99% Gen, 52.94% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 44m 16s. Estimated total time: 20h 31m 16s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 2s, 500 more iterations: 3h 25m 12s.
[2025-11-13 11:53:48,623][__main__][INFO] - Starting iteration 583.
[2025-11-13 11:53:48,626][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:53:48,626][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:58,708][__main__][INFO] - Number of regex retries in iteration 583: 0
[2025-11-13 11:53:58,709][__main__][INFO] - agents played in iteration 583 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:53:59,163][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:59,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:59,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:59,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:59,262][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:59,262][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:59,969][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:54:00,263][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:54:00,587][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:54:00,909][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:54:01,233][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:54:01,555][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:54:01,878][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:54:02,201][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:54:02,526][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:02,848][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:03,170][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:03,492][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:03,815][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:04,137][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:04,460][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:04,783][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:05,105][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:05,427][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:05,753][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:06,077][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:06,402][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:06,728][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:07,052][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:07,379][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:07,702][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:08,350][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:08,673][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:08,997][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:09,323][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:09,648][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:09,971][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:10,294][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:11,010][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:11,698][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:11,702][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:11,706][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:12,630][__main__][INFO] - Iteration 584 took 24s (42.00% Gen, 54.15% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 12m 50s. Estimated total time: 20h 0m 14s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 2s.
[2025-11-13 11:54:12,631][__main__][INFO] - Starting iteration 584.
[2025-11-13 11:54:12,634][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:12,634][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:54:21,538][__main__][INFO] - Number of regex retries in iteration 584: 0
[2025-11-13 11:54:21,538][__main__][INFO] - agents played in iteration 584 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:54:21,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:22,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:22,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:22,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:22,087][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:54:22,088][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:54:22,779][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:54:23,075][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:54:23,400][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:54:23,724][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:54:24,048][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:54:24,370][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:54:24,699][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:54:25,021][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:54:25,344][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:25,667][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:25,989][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:26,313][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:26,636][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:26,958][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:27,281][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:27,604][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:27,929][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:28,252][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:28,574][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:28,897][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:29,220][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:29,544][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:29,867][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:30,190][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:30,513][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:30,836][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:31,161][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:31,487][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:31,812][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:32,136][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:32,459][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:32,783][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:33,108][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:33,818][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:34,516][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:34,517][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:34,518][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:35,478][__main__][INFO] - Iteration 585 took 22s (38.97% Gen, 56.82% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 14m 29s. Estimated total time: 19h 2m 15s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 4s, 500 more iterations: 3h 10m 22s.
[2025-11-13 11:54:35,480][__main__][INFO] - Starting iteration 585.
[2025-11-13 11:54:35,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:35,483][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:54:45,499][__main__][INFO] - Number of regex retries in iteration 585: 0
[2025-11-13 11:54:45,499][__main__][INFO] - agents played in iteration 585 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:54:45,938][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:45,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:46,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:46,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:46,040][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:54:46,041][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:54:46,758][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:54:47,052][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:54:47,376][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:54:47,700][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:54:48,025][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:54:48,348][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:54:48,670][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:54:48,992][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:54:49,315][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:49,637][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:49,961][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:50,283][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:50,605][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:50,928][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:51,252][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:51,575][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:51,898][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:52,222][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:52,546][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:52,871][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:53,194][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:53,518][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:53,843][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:54,167][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:54,493][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:54,822][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:55,146][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:55,471][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:55,795][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:56,119][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:56,442][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:56,766][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:57,090][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:57,811][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:58,504][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:58,508][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:58,512][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:59,448][__main__][INFO] - Iteration 586 took 23s (41.79% Gen, 54.30% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 10m 6s. Estimated total time: 19h 58m 17s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 56s, 500 more iterations: 3h 19m 42s.
[2025-11-13 11:54:59,450][__main__][INFO] - Starting iteration 586.
[2025-11-13 11:54:59,453][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:59,453][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:09,472][__main__][INFO] - Number of regex retries in iteration 586: 0
[2025-11-13 11:55:09,473][__main__][INFO] - agents played in iteration 586 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:55:09,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:09,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:09,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:10,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:10,011][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:10,011][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:10,724][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:11,019][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:11,343][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:11,666][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:11,988][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:12,310][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:55:12,634][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:55:12,956][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:55:13,278][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:55:13,601][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:55:13,924][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:55:14,247][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:55:14,571][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:55:14,896][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:55:15,219][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:55:15,548][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:55:15,870][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:55:16,194][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:55:16,518][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:55:16,842][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:55:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:55:17,490][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:55:17,814][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:55:18,138][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:55:18,462][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:55:18,790][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:55:19,115][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:55:19,438][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:55:19,763][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:55:20,089][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:55:20,414][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:55:20,737][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:55:21,061][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:55:21,788][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:55:22,488][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:55:22,490][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:55:22,491][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:55:23,433][__main__][INFO] - Iteration 587 took 23s (41.78% Gen, 54.29% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 10m 28s. Estimated total time: 19h 59m 2s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 58s, 500 more iterations: 3h 19m 50s.
[2025-11-13 11:55:23,435][__main__][INFO] - Starting iteration 587.
[2025-11-13 11:55:23,438][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:55:23,439][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:33,448][__main__][INFO] - Number of regex retries in iteration 587: 0
[2025-11-13 11:55:33,448][__main__][INFO] - agents played in iteration 587 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:55:33,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:33,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:33,950][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:33,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:33,983][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:33,984][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:34,691][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:35,313][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:35,638][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:35,959][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:36,282][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:55:36,605][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:55:36,928][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:55:37,252][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:55:37,579][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:55:37,902][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:55:38,225][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:55:38,547][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:55:38,871][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:55:39,195][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:55:39,518][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:55:39,843][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:55:40,167][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:55:40,491][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:55:40,815][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:55:41,139][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:55:41,462][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:55:41,787][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:55:42,110][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:55:42,433][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:55:42,758][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:55:43,082][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:55:43,406][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:55:43,729][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:55:44,052][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:55:44,375][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:55:44,699][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:55:45,023][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:55:45,726][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:55:46,424][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:55:46,425][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:55:46,427][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:55:47,474][__main__][INFO] - Iteration 588 took 24s (41.64% Gen, 53.99% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 12m 52s. Estimated total time: 20h 1m 50s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 18s.
[2025-11-13 11:55:47,476][__main__][INFO] - Starting iteration 588.
[2025-11-13 11:55:47,479][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:55:47,480][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:56,880][__main__][INFO] - Number of regex retries in iteration 588: 0
[2025-11-13 11:55:56,880][__main__][INFO] - agents played in iteration 588 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:55:57,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:57,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:57,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:57,410][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:57,411][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:57,411][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:58,121][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:58,417][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:58,741][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:59,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:59,388][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:59,711][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:56:00,033][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:56:00,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:56:00,685][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:56:01,009][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:56:01,333][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:56:01,658][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:56:01,982][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:56:02,307][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:56:02,632][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:56:02,958][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:56:03,283][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:03,606][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:03,932][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:04,255][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:04,579][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:04,903][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:05,228][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:05,551][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:05,876][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:06,200][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:06,524][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:06,848][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:07,173][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:07,497][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:07,821][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:08,145][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:08,470][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:09,192][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:09,887][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:09,890][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:09,892][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:10,735][__main__][INFO] - Iteration 589 took 23s (40.42% Gen, 55.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 33m 28s. Estimated total time: 19h 22m 49s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 45s, 500 more iterations: 3h 13m 48s.
[2025-11-13 11:56:10,737][__main__][INFO] - Starting iteration 589.
[2025-11-13 11:56:10,739][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:56:10,740][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:56:20,262][__main__][INFO] - Number of regex retries in iteration 589: 0
[2025-11-13 11:56:20,263][__main__][INFO] - agents played in iteration 589 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:56:20,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:20,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:20,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:20,800][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:20,800][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:56:20,800][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:56:21,478][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:56:21,773][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:56:22,097][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:56:22,425][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:56:22,751][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:56:23,076][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:56:23,399][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:56:23,723][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:56:24,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:56:24,372][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:56:24,699][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:56:25,022][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:56:25,350][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:56:25,675][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:56:25,999][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:56:26,322][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:56:26,645][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:26,969][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:27,293][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:27,616][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:27,941][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:28,266][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:28,590][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:28,914][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:29,237][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:29,561][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:29,885][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:30,208][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:30,533][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:30,860][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:31,185][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:31,509][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:31,833][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:32,546][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:33,246][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:33,251][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:33,253][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:34,230][__main__][INFO] - Iteration 590 took 23s (40.53% Gen, 55.30% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 44m 50s. Estimated total time: 19h 34m 36s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 9s, 500 more iterations: 3h 15m 46s.
[2025-11-13 11:56:34,232][__main__][INFO] - Starting iteration 590.
[2025-11-13 11:56:34,235][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:56:34,236][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:56:43,279][__main__][INFO] - Number of regex retries in iteration 590: 0
[2025-11-13 11:56:43,280][__main__][INFO] - agents played in iteration 590 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:56:43,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:43,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:43,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:43,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:43,831][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:56:43,832][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:56:44,510][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:56:44,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:56:45,129][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:56:45,454][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:56:45,778][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:56:46,101][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:56:46,425][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:56:46,752][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:56:47,075][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:56:47,400][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:56:47,722][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:56:48,045][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:56:48,369][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:56:48,693][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:56:49,015][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:56:49,340][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:56:49,664][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:49,989][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:50,314][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:50,638][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:50,962][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:51,287][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:51,616][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:51,940][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:52,263][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:52,586][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:52,910][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:53,557][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:53,880][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:54,204][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:54,527][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:54,851][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:55,558][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:56,261][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:56,263][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:56,265][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:58,368][__main__][INFO] - Iteration 591 took 24s (37.47% Gen, 53.81% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 16m 32s. Estimated total time: 20h 6m 41s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 13s, 500 more iterations: 3h 21m 6s.
[2025-11-13 11:56:58,370][__main__][INFO] - Starting iteration 591.
[2025-11-13 11:56:58,373][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:56:58,374][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:07,863][__main__][INFO] - Number of regex retries in iteration 591: 0
[2025-11-13 11:57:07,864][__main__][INFO] - agents played in iteration 591 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:57:08,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:08,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:08,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:08,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:08,410][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:08,410][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:09,102][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:09,397][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:09,721][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:10,045][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:10,370][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:10,694][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:11,019][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:11,344][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:11,668][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:11,996][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:12,320][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:12,643][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:12,967][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:57:13,290][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:57:13,614][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:57:13,936][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:57:14,261][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:57:14,586][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:57:14,909][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:57:15,233][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:57:15,558][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:57:15,881][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:57:16,205][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:57:16,528][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:57:16,850][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:57:17,173][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:57:17,497][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:57:17,820][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:57:18,147][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:57:18,470][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:57:18,793][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:57:19,116][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:57:19,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:57:20,165][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:57:20,863][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:57:20,864][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:57:20,866][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:57:21,850][__main__][INFO] - Iteration 592 took 23s (40.42% Gen, 55.38% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 43m 20s. Estimated total time: 19h 33m 52s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 7s, 500 more iterations: 3h 15m 38s.
[2025-11-13 11:57:21,852][__main__][INFO] - Starting iteration 592.
[2025-11-13 11:57:21,855][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:57:21,855][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:31,505][__main__][INFO] - Number of regex retries in iteration 592: 0
[2025-11-13 11:57:31,506][__main__][INFO] - agents played in iteration 592 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:57:31,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:31,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:32,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:32,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:32,046][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:32,047][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:32,760][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:33,056][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:33,380][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:33,708][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:34,033][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:34,356][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:34,680][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:35,003][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:35,327][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:35,649][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:35,973][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:36,297][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:36,623][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:57:36,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:57:37,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:57:37,595][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:57:37,917][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:57:38,240][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:57:38,562][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:57:38,886][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:57:39,210][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:57:39,533][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:57:39,857][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:57:40,180][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:57:40,505][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:57:40,828][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:57:41,152][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:57:41,475][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:57:41,799][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:57:42,126][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:57:42,450][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:57:42,773][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:57:43,098][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:57:43,812][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:57:44,529][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:57:44,531][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:57:44,533][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:57:45,406][__main__][INFO] - Iteration 593 took 23s (40.98% Gen, 55.31% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 46m 39s. Estimated total time: 19h 37m 35s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 15s.
[2025-11-13 11:57:45,408][__main__][INFO] - Starting iteration 593.
[2025-11-13 11:57:45,411][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:57:45,412][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:54,666][__main__][INFO] - Number of regex retries in iteration 593: 0
[2025-11-13 11:57:54,667][__main__][INFO] - agents played in iteration 593 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:57:55,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:55,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:55,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:55,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:55,217][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:55,218][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:55,924][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:56,220][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:56,544][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:56,869][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:57,194][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:57,520][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:57,843][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:58,166][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:58,491][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:58,814][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:59,137][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:59,460][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:59,786][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:58:00,109][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:58:00,432][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:58:00,757][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:58:01,081][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:58:01,404][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:58:01,729][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:58:02,053][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:58:02,378][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:58:02,701][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:58:03,025][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:58:03,347][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:58:03,671][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:58:03,995][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:58:04,319][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:58:04,644][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:58:04,967][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:58:05,290][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:58:05,614][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:58:05,937][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:58:06,262][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:06,980][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:07,701][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:07,702][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:07,706][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:08,750][__main__][INFO] - Iteration 594 took 23s (39.65% Gen, 55.87% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 35m 39s. Estimated total time: 19h 26m 59s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 29s.
[2025-11-13 11:58:08,752][__main__][INFO] - Starting iteration 594.
[2025-11-13 11:58:08,756][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:08,757][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:58:17,948][__main__][INFO] - Number of regex retries in iteration 594: 0
[2025-11-13 11:58:17,948][__main__][INFO] - agents played in iteration 594 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:58:18,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:18,425][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:18,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:18,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:18,491][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:58:18,492][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:58:19,222][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:58:19,519][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:58:19,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:58:20,169][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:58:20,492][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:58:20,815][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:58:21,140][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:58:21,462][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:58:21,786][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:58:22,109][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:58:22,432][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:58:22,756][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:58:23,080][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:58:23,403][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:58:23,728][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:58:24,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:58:24,376][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:58:24,701][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:58:25,024][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:58:25,347][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:58:25,671][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:58:25,994][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:58:26,317][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:58:26,641][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:58:26,964][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:58:27,287][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:58:27,610][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:58:27,933][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:58:28,256][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:58:28,580][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:58:28,904][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:58:29,227][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:58:29,551][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:30,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:31,004][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:31,006][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:31,007][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:32,073][__main__][INFO] - Iteration 595 took 23s (39.42% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 34m 11s. Estimated total time: 19h 25m 54s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 51s, 500 more iterations: 3h 14m 19s.
[2025-11-13 11:58:32,075][__main__][INFO] - Starting iteration 595.
[2025-11-13 11:58:32,079][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:32,080][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:58:41,404][__main__][INFO] - Number of regex retries in iteration 595: 0
[2025-11-13 11:58:41,404][__main__][INFO] - agents played in iteration 595 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:58:41,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:41,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:41,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:41,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:41,951][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:58:41,951][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:58:42,671][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:58:42,966][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:58:43,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:58:43,615][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:58:43,938][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:58:44,261][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:58:44,585][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:58:44,907][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:58:45,230][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:58:45,554][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:58:45,877][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:58:46,202][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:58:46,527][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:58:46,851][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:58:47,175][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:58:47,501][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:58:47,826][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:58:48,153][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:58:48,476][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:58:48,800][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:58:49,124][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:58:49,449][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:58:49,773][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:58:50,096][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:58:50,418][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:58:50,743][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:58:51,069][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:58:51,391][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:58:51,714][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:58:52,038][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:58:52,362][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:58:52,686][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:58:53,008][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:53,699][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:54,435][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:54,436][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:54,441][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:55,509][__main__][INFO] - Iteration 596 took 23s (39.80% Gen, 55.64% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 39m 24s. Estimated total time: 19h 31m 31s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 3s, 500 more iterations: 3h 15m 15s.
[2025-11-13 11:58:55,511][__main__][INFO] - Starting iteration 596.
[2025-11-13 11:58:55,514][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:55,515][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:59:04,605][__main__][INFO] - Number of regex retries in iteration 596: 0
[2025-11-13 11:59:04,606][__main__][INFO] - agents played in iteration 596 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:59:05,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:05,089][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:05,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:05,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:05,156][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:59:05,156][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:59:05,872][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:59:06,167][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:59:06,490][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:59:06,815][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:59:07,138][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:07,464][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:07,787][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:08,110][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:08,433][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:08,756][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:09,081][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:09,404][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:09,728][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:10,051][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:10,375][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:10,699][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:11,022][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:11,351][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:11,677][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:12,005][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:12,331][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:59:12,657][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:59:12,982][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:59:13,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:59:13,631][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:59:13,956][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:59:14,279][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:59:14,603][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:59:14,930][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:59:15,254][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:59:15,578][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:59:15,902][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:59:16,225][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:59:16,911][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:59:17,654][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:59:17,655][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:59:17,657][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:59:18,648][__main__][INFO] - Iteration 597 took 23s (39.29% Gen, 56.42% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 24m 14s. Estimated total time: 19h 16m 43s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 47s.
[2025-11-13 11:59:18,650][__main__][INFO] - Starting iteration 597.
[2025-11-13 11:59:18,653][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:59:18,653][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:59:28,164][__main__][INFO] - Number of regex retries in iteration 597: 0
[2025-11-13 11:59:28,165][__main__][INFO] - agents played in iteration 597 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:59:28,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:28,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:28,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:28,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:28,730][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:59:28,730][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:59:29,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:59:29,737][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:59:30,063][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:59:30,387][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:59:30,711][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:31,036][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:31,360][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:31,684][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:32,007][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:32,331][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:32,655][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:32,979][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:33,303][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:33,628][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:33,952][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:34,277][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:34,601][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:34,925][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:35,248][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:35,573][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:35,897][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:59:36,222][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:59:36,547][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:59:36,870][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:59:37,196][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:59:37,519][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:59:37,843][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:59:38,166][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:59:38,489][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:59:38,814][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:59:39,138][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:59:39,462][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:59:39,788][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:59:40,508][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:59:41,233][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:59:41,235][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:59:41,236][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:59:42,611][__main__][INFO] - Iteration 598 took 23s (39.70% Gen, 54.56% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 5m 2s. Estimated total time: 19h 57m 56s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 55s, 500 more iterations: 3h 19m 39s.
[2025-11-13 11:59:42,612][__main__][INFO] - Starting iteration 598.
[2025-11-13 11:59:42,616][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:59:42,616][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:59:52,242][__main__][INFO] - Number of regex retries in iteration 598: 0
[2025-11-13 11:59:52,243][__main__][INFO] - agents played in iteration 598 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 11:59:52,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:52,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:52,769][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:52,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:52,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:59:52,804][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:59:53,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:59:53,814][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:59:54,139][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:59:54,462][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:59:54,786][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:55,110][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:55,431][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:55,754][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:56,078][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:56,402][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:56,725][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:57,049][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:57,373][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:57,695][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:58,019][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:58,343][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:58,668][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:58,991][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:59,314][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:59,638][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:59,961][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:00:00,287][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:00:00,611][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:00:00,934][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:00:01,259][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:00:01,582][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:00:01,907][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:00:02,233][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:00:02,557][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:00:02,883][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:00:03,209][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:00:03,533][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:00:03,858][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:00:04,571][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:00:05,307][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:00:05,308][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:00:05,310][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:00:06,832][__main__][INFO] - Iteration 599 took 24s (39.75% Gen, 53.96% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 17m 34s. Estimated total time: 20h 10m 52s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 21s, 500 more iterations: 3h 21m 48s.
[2025-11-13 12:00:06,834][__main__][INFO] - Starting iteration 599.
[2025-11-13 12:00:06,837][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 12:00:06,838][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:00:15,816][__main__][INFO] - Number of regex retries in iteration 599: 0
[2025-11-13 12:00:15,817][__main__][INFO] - agents played in iteration 599 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:00:16,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:16,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:16,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:16,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:16,375][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:00:16,376][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:00:17,100][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:00:17,395][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:00:17,719][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:00:18,044][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:00:18,367][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:00:18,691][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:00:19,015][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:00:19,340][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:00:19,665][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:00:19,989][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:00:20,312][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:00:20,635][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:00:20,959][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:00:21,284][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:00:21,608][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:00:21,932][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:00:22,256][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:00:22,580][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:00:22,905][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:00:23,231][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:00:23,556][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:00:23,880][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:00:24,205][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:00:24,529][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:00:24,854][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:00:25,178][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:00:25,503][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:00:25,827][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:00:26,151][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:00:26,476][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:00:26,801][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:00:27,124][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:00:27,447][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:00:28,133][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:00:28,862][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:00:28,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:00:28,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:00:30,245][__main__][INFO] - Iteration 600 took 23s (38.36% Gen, 55.74% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 36m 45s. Estimated total time: 19h 30m 26s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 0s, 500 more iterations: 3h 15m 4s.
[2025-11-13 12:00:30,248][__main__][INFO] - Starting iteration 600.
[2025-11-13 12:00:30,251][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 12:00:30,252][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:00:39,080][__main__][INFO] - Number of regex retries in iteration 600: 0
[2025-11-13 12:00:39,081][__main__][INFO] - agents played in iteration 600 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:00:39,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:39,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:39,601][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:39,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:39,636][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:00:39,636][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:00:40,367][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:00:40,663][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:00:40,986][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:00:41,309][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:00:41,632][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:00:41,955][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:00:42,279][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:00:42,603][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:00:42,927][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:00:43,256][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:00:43,584][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:00:43,907][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:00:44,231][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:00:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:00:44,880][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:00:45,203][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:00:45,527][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:00:45,850][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:00:46,175][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:00:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:00:46,823][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:00:47,148][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:00:47,471][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:00:47,795][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:00:48,120][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:00:48,444][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:00:48,769][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:00:49,092][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:00:49,417][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:00:49,741][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:00:50,065][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:00:50,387][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:00:50,711][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:00:51,399][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:00:52,125][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:00:52,127][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:00:52,130][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:00:54,629][__main__][INFO] - Iteration 601 took 24s (36.22% Gen, 53.53% Train). Generation: 8s, Training: 13s. Estimated remaining time: 19h 24m 50s. Estimated total time: 20h 18m 55s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 37s, 500 more iterations: 3h 23m 9s.
[2025-11-13 12:00:54,631][__main__][INFO] - Starting iteration 601.
[2025-11-13 12:00:54,634][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:00:54,635][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:01:04,337][__main__][INFO] - Number of regex retries in iteration 601: 0
[2025-11-13 12:01:04,338][__main__][INFO] - agents played in iteration 601 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:01:04,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:04,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:04,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:04,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:04,896][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:01:04,897][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:01:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:01:05,915][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:01:06,242][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:01:06,565][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:01:06,888][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:01:07,212][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:01:07,537][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:01:07,865][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:01:08,189][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:01:08,513][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:01:08,837][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:01:09,161][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:01:09,486][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:01:09,810][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:01:10,133][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:01:10,456][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:01:10,780][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:11,106][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:11,433][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:11,757][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:12,083][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:12,406][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:12,730][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:13,053][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:01:13,377][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:01:13,700][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:01:14,028][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:01:14,352][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:01:14,676][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:01:14,998][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:01:15,322][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:01:15,647][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:01:15,970][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:01:16,656][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:01:17,375][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:01:17,376][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:01:17,378][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:01:18,559][__main__][INFO] - Iteration 602 took 23s (40.55% Gen, 54.50% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 1m 46s. Estimated total time: 19h 56m 15s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 52s, 500 more iterations: 3h 19m 22s.
[2025-11-13 12:01:18,560][__main__][INFO] - Starting iteration 602.
[2025-11-13 12:01:18,564][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:01:18,564][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:01:27,894][__main__][INFO] - Number of regex retries in iteration 602: 0
[2025-11-13 12:01:27,895][__main__][INFO] - agents played in iteration 602 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:01:28,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:28,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:28,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:28,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:28,454][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:01:28,455][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:01:29,215][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:01:29,512][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:01:29,841][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:01:30,164][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:01:30,488][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:01:30,813][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:01:31,136][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:01:31,459][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:01:31,784][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:01:32,107][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:01:32,431][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:01:32,757][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:01:33,080][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:01:33,403][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:01:33,727][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:01:34,051][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:01:34,377][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:34,702][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:35,026][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:35,354][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:35,678][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:36,002][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:36,325][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:36,648][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:01:36,972][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:01:37,294][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:01:37,619][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:01:37,942][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:01:38,266][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:01:38,589][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:01:38,913][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:01:39,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:01:39,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:01:40,284][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:01:41,022][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:01:41,023][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:01:41,025][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:01:42,090][__main__][INFO] - Iteration 603 took 23s (39.66% Gen, 55.81% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 41m 29s. Estimated total time: 19h 36m 23s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 3s.
[2025-11-13 12:01:42,092][__main__][INFO] - Starting iteration 603.
[2025-11-13 12:01:42,096][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:01:42,096][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:01:51,232][__main__][INFO] - Number of regex retries in iteration 603: 0
[2025-11-13 12:01:51,233][__main__][INFO] - agents played in iteration 603 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:01:51,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:51,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:51,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:51,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:51,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:01:51,807][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:01:52,547][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:01:52,843][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:01:53,168][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:01:53,493][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:01:53,818][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:01:54,142][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:01:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:01:54,793][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:01:55,117][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:01:55,441][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:01:55,765][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:01:56,089][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:01:56,414][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:01:56,738][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:01:57,062][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:01:57,386][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:01:57,710][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:58,033][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:58,359][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:58,682][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:59,005][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:59,329][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:59,651][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:59,974][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:00,298][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:00,622][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:00,944][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:01,267][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:01,589][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:01,912][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:02,236][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:02,559][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:02,884][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:02:03,576][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:02:04,292][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:02:04,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:02:04,295][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:02:05,610][__main__][INFO] - Iteration 604 took 23s (38.85% Gen, 55.55% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 40m 29s. Estimated total time: 19h 35m 45s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 57s.
[2025-11-13 12:02:05,612][__main__][INFO] - Starting iteration 604.
[2025-11-13 12:02:05,615][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:02:05,615][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:02:14,740][__main__][INFO] - Number of regex retries in iteration 604: 0
[2025-11-13 12:02:14,741][__main__][INFO] - agents played in iteration 604 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:02:15,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:15,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:15,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:15,305][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:15,306][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:02:15,306][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:02:16,047][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:02:16,342][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:02:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:02:16,989][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:02:17,313][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:02:17,637][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:02:17,960][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:02:18,283][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:02:18,608][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:02:18,931][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:02:19,255][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:02:19,577][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:02:19,902][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:02:20,226][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:02:20,550][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:02:20,874][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:02:21,202][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:02:21,528][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:02:21,851][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:02:22,172][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:02:22,494][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:02:22,819][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:02:23,143][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:02:23,466][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:23,790][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:24,113][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:24,436][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:24,762][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:25,085][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:25,411][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:25,736][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:26,060][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:26,385][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:02:27,110][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:02:27,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:02:27,839][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:02:27,841][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:02:28,847][__main__][INFO] - Iteration 605 took 23s (39.28% Gen, 56.39% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 25m 59s. Estimated total time: 19h 21m 38s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 36s.
[2025-11-13 12:02:28,849][__main__][INFO] - Starting iteration 605.
[2025-11-13 12:02:28,852][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:02:28,853][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:02:38,428][__main__][INFO] - Number of regex retries in iteration 605: 0
[2025-11-13 12:02:38,428][__main__][INFO] - agents played in iteration 605 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:02:38,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:38,921][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:38,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:38,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:38,988][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:02:38,988][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:02:39,710][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:02:40,005][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:02:40,330][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:02:40,652][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:02:40,975][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:02:41,300][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:02:41,623][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:02:41,946][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:02:42,270][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:02:42,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:02:42,917][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:02:43,240][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:02:43,565][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:02:43,888][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:02:44,211][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:02:44,535][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:02:44,858][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:02:45,181][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:02:45,509][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:02:45,835][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:02:46,160][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:02:46,483][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:02:46,806][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:02:47,130][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:47,454][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:47,776][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:48,099][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:48,423][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:48,746][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:49,071][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:49,394][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:49,717][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:50,046][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:02:50,772][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:02:51,511][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:02:51,513][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:02:51,515][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:02:52,918][__main__][INFO] - Iteration 606 took 24s (39.79% Gen, 54.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 17s. Estimated total time: 20h 3m 21s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 6s, 500 more iterations: 3h 20m 33s.
[2025-11-13 12:02:52,921][__main__][INFO] - Starting iteration 606.
[2025-11-13 12:02:52,924][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:02:52,924][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:01,902][__main__][INFO] - Number of regex retries in iteration 606: 0
[2025-11-13 12:03:01,903][__main__][INFO] - agents played in iteration 606 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:03:02,367][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:02,401][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:02,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:02,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:02,469][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:02,469][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:03,199][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:03,494][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:03,820][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:03:04,143][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:03:04,468][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:03:04,791][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:03:05,115][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:03:05,440][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:03:05,764][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:03:06,087][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:03:06,413][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:03:06,737][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:03:07,061][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:03:07,386][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:03:07,709][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:03:08,037][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:03:08,360][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:03:08,682][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:03:09,005][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:03:09,328][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:03:09,651][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:03:09,977][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:03:10,305][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:03:10,630][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:03:10,956][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:03:11,280][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:03:11,604][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:03:11,928][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:03:12,253][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:03:12,576][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:03:12,899][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:03:13,222][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:03:13,544][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:03:14,213][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:03:14,939][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:03:14,940][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:03:14,942][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:03:16,013][__main__][INFO] - Iteration 607 took 23s (38.89% Gen, 56.47% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 18m 2s. Estimated total time: 19h 14m 29s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 24s.
[2025-11-13 12:03:16,015][__main__][INFO] - Starting iteration 607.
[2025-11-13 12:03:16,019][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:03:16,019][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:25,231][__main__][INFO] - Number of regex retries in iteration 607: 0
[2025-11-13 12:03:25,231][__main__][INFO] - agents played in iteration 607 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:03:25,687][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:25,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:25,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:25,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:25,788][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:25,788][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:26,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:26,814][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:27,139][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:03:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:03:27,786][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:03:28,109][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:03:28,436][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:03:28,759][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:03:29,083][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:03:29,407][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:03:29,731][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:03:30,054][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:03:30,382][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:03:30,706][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:03:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:03:31,352][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:03:31,675][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:03:31,999][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:03:32,325][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:03:32,652][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:03:32,977][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:03:33,302][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:03:33,625][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:03:33,949][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:03:34,272][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:03:34,600][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:03:34,929][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:03:35,255][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:03:35,583][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:03:35,912][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:03:36,237][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:03:36,561][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:03:36,886][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:03:37,557][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:03:38,281][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:03:38,282][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:03:38,284][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:03:39,311][__main__][INFO] - Iteration 608 took 23s (39.55% Gen, 56.03% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 48s. Estimated total time: 19h 24m 39s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 49s, 500 more iterations: 3h 14m 6s.
[2025-11-13 12:03:39,313][__main__][INFO] - Starting iteration 608.
[2025-11-13 12:03:39,316][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:03:39,317][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:48,731][__main__][INFO] - Number of regex retries in iteration 608: 0
[2025-11-13 12:03:48,732][__main__][INFO] - agents played in iteration 608 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:03:49,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:49,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:49,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:49,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:49,289][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:49,289][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:50,031][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:50,325][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:50,650][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:03:50,973][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:03:51,295][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:03:51,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:03:51,942][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:03:52,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:03:52,588][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:03:52,910][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:03:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:03:53,557][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:03:53,880][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:03:54,205][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:03:54,529][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:03:54,853][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:03:55,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:03:55,503][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:03:55,826][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:03:56,148][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:03:56,473][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:03:56,798][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:03:57,122][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:03:57,447][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:03:57,772][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:03:58,101][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:03:58,426][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:03:58,751][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:03:59,075][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:03:59,397][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:03:59,722][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:00,045][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:00,369][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:01,036][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:04:01,753][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:01,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:01,756][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:02,789][__main__][INFO] - Iteration 609 took 23s (40.11% Gen, 55.48% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 36m 28s. Estimated total time: 19h 33m 42s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 7s, 500 more iterations: 3h 15m 37s.
[2025-11-13 12:04:02,791][__main__][INFO] - Starting iteration 609.
[2025-11-13 12:04:02,795][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:04:02,795][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:04:11,643][__main__][INFO] - Number of regex retries in iteration 609: 0
[2025-11-13 12:04:11,644][__main__][INFO] - agents played in iteration 609 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:04:12,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:12,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:12,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:12,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:12,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:04:12,203][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:04:12,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:04:13,234][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:04:13,558][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:04:13,882][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:04:14,205][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:04:14,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:04:14,851][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:04:15,173][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:04:15,497][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:04:15,819][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:04:16,143][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:04:16,466][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:04:16,791][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:04:17,114][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:04:17,439][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:04:17,761][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:04:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:04:18,409][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:04:18,735][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:04:19,060][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:04:19,383][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:04:19,708][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:04:20,036][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:04:20,359][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:04:20,685][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:04:21,008][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:04:21,336][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:04:21,660][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:04:21,985][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:04:22,308][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:04:22,631][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:22,956][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:23,281][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:23,955][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:04:24,671][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:24,673][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:24,674][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:25,675][__main__][INFO] - Iteration 610 took 22s (38.67% Gen, 56.95% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 6m 28s. Estimated total time: 19h 4m 5s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 8s, 500 more iterations: 3h 10m 40s.
[2025-11-13 12:04:25,677][__main__][INFO] - Starting iteration 610.
[2025-11-13 12:04:25,681][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:04:25,681][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:04:34,952][__main__][INFO] - Number of regex retries in iteration 610: 0
[2025-11-13 12:04:34,953][__main__][INFO] - agents played in iteration 610 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:04:35,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:35,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:35,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:35,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:35,509][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:04:35,509][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:04:36,243][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:04:36,538][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:04:36,863][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:04:37,188][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:04:37,511][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:04:37,834][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:04:38,158][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:04:38,482][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:04:38,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:04:39,129][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:04:39,452][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:04:39,777][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:04:40,100][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:04:40,423][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:04:40,747][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:04:41,070][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:04:41,392][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:04:41,715][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:04:42,037][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:04:42,360][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:04:42,684][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:04:43,008][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:04:43,331][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:04:43,654][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:04:43,977][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:04:44,300][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:04:44,623][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:04:44,948][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:04:45,271][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:04:45,594][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:04:45,917][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:46,241][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:46,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:47,235][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:10
[2025-11-13 12:04:47,952][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:47,953][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:47,955][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:50,619][__main__][INFO] - Iteration 611 took 24s (37.17% Gen, 52.14% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 48m 56s. Estimated total time: 20h 46m 57s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 33s, 500 more iterations: 3h 27m 49s.
[2025-11-13 12:04:50,621][__main__][INFO] - Starting iteration 611.
[2025-11-13 12:04:50,625][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:04:50,625][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:04:59,669][__main__][INFO] - Number of regex retries in iteration 611: 0
[2025-11-13 12:04:59,670][__main__][INFO] - agents played in iteration 611 are Bob, Bob_buffer, Alice, Alice_buffer
[2025-11-13 12:05:00,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:05:00,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:05:00,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:05:00,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:05:00,222][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:05:00,222][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:05:00,955][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:05:01,250][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:05:01,575][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:05:01,899][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:05:02,223][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:05:02,546][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:05:02,870][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:05:03,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:05:03,516][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:05:03,839][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:05:04,163][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:05:04,486][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:05:04,811][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:05:05,139][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:05:05,467][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:05:05,792][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:05:06,117][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:05:06,442][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:05:06,764][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:05:07,087][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:05:07,410][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:05:07,738][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:05:08,067][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:05:08,390][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:05:08,715][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:05:09,041][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:05:09,364][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:05:09,688][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:05:10,012][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:05:10,337][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:05:10,664][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:05:10,992][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:05:11,317][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:05:12,467][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:05:13,221][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:05:13,223][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:05:13,225][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1337_bs128/seed_1337/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:05:14,694][__main__][INFO] - Iteration 612 took 24s (37.58% Gen, 56.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 5m 4s. Estimated total time: 20h 3m 30s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 7s, 500 more iterations: 3h 20m 35s.
[2025-11-13 12:05:14,696][__main__][INFO] - Starting iteration 612.
[2025-11-13 12:05:17,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,768][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,768][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,768][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,768][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,768][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,769][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,770][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,770][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,770][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,770][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,770][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,771][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,772][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,772][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,772][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,772][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,772][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,773][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,774][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,774][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,774][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,774][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,774][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,775][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,776][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,777][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,778][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,779][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,780][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,781][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,782][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,783][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,784][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,785][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,786][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,787][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,788][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,789][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,790][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,790][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,790][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,790][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,790][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,830][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,830][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,831][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,832][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,833][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,834][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,835][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,836][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,837][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,838][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,839][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,840][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,841][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,842][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,842][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,842][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,842][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,842][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,843][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,844][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:17,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,127][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,128][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,134][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,135][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,136][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,137][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,142][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,143][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,144][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,145][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,146][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,147][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,148][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,149][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,150][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,151][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,152][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,153][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,154][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,155][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,156][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,157][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,158][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,159][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,160][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,161][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,162][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,163][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,164][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,165][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,166][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,167][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,168][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,169][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,170][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,171][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,172][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,173][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,174][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,175][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,176][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,177][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,178][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,179][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,180][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,181][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,182][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,183][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,184][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,185][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,186][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,187][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,188][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,189][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,190][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,191][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,192][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,193][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,194][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,195][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,196][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,197][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,198][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,199][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,200][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,201][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,202][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,203][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,204][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,205][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,206][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,207][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,632][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,633][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,634][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,635][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,636][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,637][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,638][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,639][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,640][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,641][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,642][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,643][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,644][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,645][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,646][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,647][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,648][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,649][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,650][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,651][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,652][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,653][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,654][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,655][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,656][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,657][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,658][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,659][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,660][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,661][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,662][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,663][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,664][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:18,767][asyncio][WARNING] - socket.send() raised exception.